Emerging cyber technology permeates not only our working environment but also our daily life. We enjoy various cyber-enabled online services, e.g., using smartphones for navigation and friendships via social network services (SNS) and doing business through the cyberspace, where a large amount of human-centric data are produced in both the man–man and man–machine systems. This issue aims to enhance the next generation of human-centered social computing and provide relevant theoretical and algorithmic support for the application of social networks. Accepted articles included in this special issue cover topics on human-centric big data analysis, social influence analysis, data-driven interdisciplinary modeling and analysis, user behavior modeling and analysis, cyber-enabled and artificial intelligent (AI)-enhanced healthcare services, and so on. These contents may draft promising perspectives for future research and demonstrate a variety of issues and solutions related to social computing. With the AI-enhanced technical analysis, these human-centric data can be effectively used to better serve humanity.

Annotating a data sample in a multi-label learning problem requires a human oracle to consider the presence/absence of every possible label separately, which is extremely labor intensive. Active learning algorithms automatically identify the informative samples from large amounts of unlabeled data and significantly reduce human annotation efforts in inducing a classification model. Further, deep models have gained popularity to automatically learn representative features from a given dataset and have depicted promising empirical performance in a variety of applications. In this paper, we exploit the feature learning capabilities of deep neural networks and propose a novel framework to address the problem of multi-label active learning with label correlation. We integrate an active selection criterion to the objective function and train deep networks to optimize the function. Our extensive empirical studies on five benchmark multi-label datasets show that our methods outperform the state-of-the-art algorithms, corroborating their potential for real-world image classification applications.

In recent years, deep neural networks have emerged as a dominant machine learning tool for a wide variety of application domains. However, training a deep neural network requires a large amount of labeled data, which is an expensive process in terms of time, labor and human expertise. Domain adaptation or transfer learning algorithms address this challenge by leveraging labeled data in a different, but related source domain, to develop a model for the target domain. Further, the explosive growth of digital data has posed a fundamental challenge concerning its storage and retrieval. Due to its storage and retrieval efficiency, recent years have witnessed a wide application of hashing in a variety of computer vision applications. In this paper, we first introduce a new dataset, Office-Home, to evaluate domain adaptation algorithms. The dataset contains images of a variety of everyday objects from multiple domains. We then propose a novel deep learning framework that can exploit labeled source data and unlabeled target data to learn informative hash codes, to accurately classify unseen target data. To the best of our knowledge, this is the first research effort to exploit the feature learning capabilities of deep neural networks to learn representative hash codes to address the domain adaptation problem. Our extensive empirical studies on multiple transfer tasks corroborate the usefulness of the framework in learning efficient hash codes which outperform existing competitive baselines for unsupervised domain adaptation.

In the recent years, deep learning algorithms have achieved state-of-the-art performance in a variety of computer vision applications. In this paper, we propose a novel active learning framework to select the most informative unlabeled samples to train a deep belief network model. We introduce a loss function specific to the active learning task and train the model to minimize the loss function. To the best of our knowledge, this is the first research effort to integrate an active learning based criterion in the loss function used to train a deep belief network. Our extensive empirical studies on a wide variety of uni-modal and multi-modal vision datasets corroborate the potential of the method for real-world image recognition applications.

This paper presents the design of TranslatAble, a new Augmentative and Alternative Communication system. Taking advantage of the convenience and mobility of modern smartphones, TranslatAble allows individuals with complex communication needs-specifically individuals with severe dysarthria or individuals who primarily use non-sign language gestures---to build their own system of communication through speech or gesture input. This system uses a machine learning model---specifically dynamic time warping---to match the user's input to their unique dictionary to help them be understood by their communication partner. TranslatAble is highly flexible and person-centered, which allows it to adapt to fit the unique needs of each user's unique communication needs. We expect this novel device to enable individuals with complex communication needs to interact with a larger social circle. This may help decrease feelings of loneliness and increase self-confidence, self-esteem, and independence, as well as help maintain strong relationships outside of the user's support network.

We present a novel framework and system for at-home rehabilitative exercise in the absence of a physical therapist. The framework includes metrics for assessing motor performance on a wide variety of exercises. We present our system, the Autonomous Training Assistant, which utilizes this framework and a low-cost accessible exercise device called the Intelligent Stick to deliver feedback as a user trains at home. We evaluated the system's multimodal feedback mechanism in a case study whose results indicate that individual preference may have a significant effect on modality assignment for optimal learning. We conclude with ideas for future work.

With the appropriate mathematical models, data from wearable devices can be used to help Parkinson's patients live safer and more independent lives. Inspired by this idea, the purpose of this study was to determine the viability of neural networks in predicting Freezing of Gait (FoG), a symptom of Parkinson's disease in which the patient's legs are suddenly rendered unable to move. A class of neural networks known as layered recurrent networks (LRNs) was applied to an open-source FoG experimental dataset donated to the Machine Learning Repository of the University of California at Irvine. The independent variables in this experiment -- the subject being tested, neural network architecture, and down sampling of the majority classes -- were each varied and compared against the performance of the neural network in predicting impending FoG events. It was determined that single-layered recurrent networks are a viable method of predicting FoG events given the volume of the training data available, though results varied between patients.

Rapid urbanization has led to more people residing in cities than ever before, and projections estimate that 64% of the global population will be urban by 2050. Cities are beginning to explore Smart City initiatives to reduce expenses and complexities while increasing efficiency and quality of life for its citizens. To achieve this goal, advances in technology and policies are needed together with rethinking traditional solutions to transportation, safety, sustainability, among other priority areas. We propose the use of a Smart Stadium as a 'living laboratory' to identify, deploy and test Internet of Things technologies and Smart City solutions in an environment small enough to practically trial but large enough to evaluate effectiveness and scalability. The Smart Stadium for Smarter Living initiative brings together Arizona State University, Dublin City University, Intel Corporation, Gaelic Athletic Association, Sun Devil Stadium and Croke Park to explore smart environment solutions.

In this paper we introduce a novel platform that utilises multi-modal low-cost motion capture technology for the delivery of real-time visual feedback for sports performance. This platform supports the expansion to multi-modal interfaces that utilise haptic and audio feedback, which scales effectively with motor task complexity. We demonstrate an implementation of our platform within the field of sports performance. The platform includes low-cost motion capture through a fusion technique, combining a Microsoft Kinect V2 with two wrist inertial sensors, which make use of the accelerometer and gyroscope sensors, alongside a game-based Graphical User Interface (GUI) for instruction, visual feedback and gamified score tracking.

Emotion analysis and recognition has become an interesting topic of research among the computer vision research community. In this paper, we first present the emoF-BVP database of multimodal (face, body gesture, voice and physiological signals) recordings of actors enacting various expressions of emotions. The database consists of audio and video sequences of actors displaying three different intensities of expressions of 23 different emotions along with facial feature tracking, skeletal tracking and the corresponding physiological data. Next, we describe four deep belief network (DBN) models and show that these models generate robust multimodal features for emotion classification in an unsupervised manner. Our experimental results show that the DBN models perform better than the state of the art methods for emotion recognition. Finally, we propose convolutional deep belief network (CDBN) models that learn salient multimodal features of expressions of emotions. Our CDBN models give better recognition accuracies when recognizing low intensity or subtle expressions of emotions when compared to state of the art methods.

Mutual Information (MI) is often used for feature selection when developing classifier models. Estimating the MI for a subset of features is often intractable. We demonstrate, that under the assumptions of conditional independence, MI between a subset of features can be expressed as the Conditional Mutual Information (CMI) between pairs of features. But selecting features with the highest CMI turns out to be a hard combinatorial problem. In this work, we have applied two unique global methods, Truncated Power Method (TPower) and Low Rank Bilinear Approximation (LowRank), to solve the feature selection problem. These algorithms provide very good approximations to the NP-hard CMI based feature selection problem. We experimentally demonstrate the effectiveness of these procedures across multiple datasets and compare them with existing MI based global and iterative feature selection procedures.

Dimension reduction techniques form the core of predictive analytics systems and they help us create a new feature space that is more helpful in predicting response variables. But these techniques do not necessarily guarantee a better predictive capability as most of them are unsupervised, especially in regression learning. In regression analysis literature, supervised dimension reduction techniques have not been explored much and in this work we provide a solution to this through probabilistic topic models. In this work, we have shown that the double mixture structure of Latent Dirichlet Allocation topic model helps us 1) to visualize feature patterns, and 2) to project features onto a topic simplex that is more predictive of responses, when compared to popular techniques like PCA and KernelPCA. Until now, topic models have not been explored in a supervised context of video analysis and in this work we introduce a Regularized supervised topic model (RSLDA) that models video and audio features and has outperformed supervised dimension reduction techniques like SPCA and Correlation based feature selection algorithms. All the models discussed in this work have been evaluated to predict continuous human emotions from video data.

Active learning algorithms automatically identify the salient and exemplar instances from large amounts of unlabeled data and thus reduce human annotation effort in inducing a classification model. More recently, Batch Mode Active Learning (BMAL) techniques have been proposed, where a batch of data samples is selected simultaneously from an unlabeled set. Most active learning algorithms assume a flat label space, that is, they consider the class labels to be independent. However, in many applications, the set of class labels are organized in a hierarchical tree structure, with the leaf nodes as outputs and the internal nodes as clusters of outputs at multiple levels of granularity. In this paper, we propose a novel BMAL algorithm (BatchRank) for hierarchical classification. The sample selection is posed as an NP-hard integer quadratic programming problem and a convex relaxation (based on linear programming) is derived, whose solution is further improved by an iterative truncated power method. Finally, a deterministic bound is established on the quality of the solution. Our empirical results on several challenging, real-world datasets from multiple domains, corroborate the potential of the proposed framework for real-world hierarchical classification applications.

Popular domain adaptation (DA) techniques learn a classifier for the target domain by sampling relevant data points from the source and combining it with the target data. We present a Support Vector Machine (SVM) based supervised DA technique, where the similarity between source and target domains is modeled as the similarity between their SVM decision boundaries. We couple the source and target SVMs and reduce the model to a standard single SVM. We test the Coupled-SVM on multiple datasets and compare our results with other popular SVM based DA approaches.

People who are blind have the inherent tendency of exploring their interaction partner's face to know them better and feel closer. This becomes a necessity when partners become engaged in a conversation and want to be aware of the different facial expressions. Current technologies can deliver coarse-grained abstract haptic cues to the visually impaired. A better solution is needed -- one that allows a blind individual interpretation of others' facial expressions while engaged in bilateral conversation. To meet this goal, the development and evaluation of a platform called 'iHap' is proposed - an interactive haptic-based "Explore-learn-interact" paradigm that enables a blind individual to access his own facial expressions in a dynamic virtual environment. It is hypothesized that persistent haptic exploration of different movements of a blind individual's own facial features for different expressions in the virtual environment will help him master the haptic language of facial expressions. Hence he will be able to explore others' facial expressions/emotions while engaged in social interaction. As a blind individual makes active use of the sense of touch through feature by feature analysis, emphasis has been given to feature-based active exploration of facial expressions.

Active learning techniques have gained popularity to reduce human effort in labeling data instances for inducing a classifier. When faced with large amounts of unlabeled data, such algorithms automatically identify the exemplar instances for manual annotation. More recently, there have been attempts towards a batch mode form of active learning, where a batch of data points is simultaneously selected from an unlabeled set. In this paper, we propose two novel batch mode active learning (BMAL) algorithms: BatchRank and BatchRand. We first formulate the batch selection task as an NP-hard optimization problem; we then propose two convex relaxations, one based on linear programming and the other based on semi-definite programming to solve the batch selection problem. Finally, a deterministic bound is derived on the solution quality for the first relaxation and a probabilistic bound for the second. To the best of our knowledge, this is the first research effort to derive mathematical guarantees on the solution quality of the BMAL problem. Our extensive empirical studies on 15 binary, multi-class and multi-label challenging datasets corroborate that the proposed algorithms perform at par with the state-of-the-art techniques, deliver high quality solutions and are robust to real-world issues like label noise and class imbalance.

Active learning techniques have gained popularity to reduce human effort in labeling data instances for inducing a classifier. When faced with large amounts of unlabeled data, such algorithms automatically identify the exemplar and representative instances to be selected for manual annotation. More recently, there have been attempts toward a batch mode form of active learning, where a batch of data points is simultaneously selected from an unlabeled set. Real-world applications require adaptive approaches for batch selection in active learning, depending on the complexity of the data stream in question. However, the existing work in this field has primarily focused on static or heuristic batch size selection. In this paper, we propose two novel optimization-based frameworks for adaptive batch mode active learning (BMAL), where the batch size as well as the selection criteria are combined in a single formulation. We exploit gradient-descent-based optimization strategies as well as properties of submodular functions to derive the adaptive BMAL algorithms. The solution procedures have the same computational complexity as existing state-of-the-art static BMAL techniques. Our empirical results on the widely used VidTIMIT and the mobile biometric (MOBIO) data sets portray the efficacy of the proposed frameworks and also certify the potential of these approaches in being used for real-world biometric recognition applications.

Communicating different emotions to individuals who are blind through tactile feedback is an active area of research. But most work is static in nature as different facial expressions of emotions are conveyed through a fixed set of facial features which may have meaning only to those who previously had sight. To individuals who are congenitally blind, these fixed sets of information are abstract, and little research reflects how this population can properly interpret and relate these fixed sets of signs as per their own nonvisual experience. Our goal is to develop a complete system that integrates feature extraction with haptic recognition. As emotion detection through image and video analysis often fails, we give emphasis on active exploration of facial expressions of one's self so that the movement of facial features and expressions becomes meaningful to users toward becoming proficient at interpreting facial expressions related to different emotions. We propose a dynamic haptic environment where an individual who is blind can perceive the reflection of his own facial movements to better understand and explore different facial expressions on the basis of the movement of his own facial features.

Affective computing researchers have recently been focusing on continuous emotion dimensions like arousal and valence. This dual coordinate affect space can explain many of the discrete emotions like sadness, anger, joy, etc. In the area of continuous emotion recognition, Principal Component Analysis (PCA) models are generally used to enhance the performance of various image and audio features by projecting them to a new space where the new features are less correlated. We instead, propose that quantizing and projecting the features to a latent topic space performs better than PCA. Specifically we extract these topic features using Latent Dirichlet Allocation (LDA) models. We show that topic models project the original features to a latent feature space that is more coherent and useful for continuous emotion recognition than PCA. Unlike PCA where no semantics can be attributed to the new features, topic features can have a visual and semantic interpretation which can be used in personalized HCI applications and Assistive technologies. Our hypothesis in this work has been validated using the AVEC 2012 continuous emotion challenge dataset.

Recovering a matrix from a sampling of its entries is a problem of rapidly growing interest and has been studied under the name of matrix completion. It occurs in many areas of engineering and applied science. In most machine learning and data mining applications, it is possible to leverage the expertise of human oracles to improve the performance of the system. It is therefore natural to extend this idea of "human-in-the-loop" to the matrix completion problem. However, considering the enormity of data in the modern era, manually completing all the entries in a matrix will be an expensive process in terms of time, labor and human expertise, human oracles can only provide selective supervision to guide the solution process. Thus, appropriately identifying a subset of missing entries (for manual annotation) in an incomplete matrix is of paramount practical importance, this can potentially lead to better reconstructions of the incomplete matrix with minimal human effort. In this paper, we propose novel algorithms to address this issue. Since the query locations are actively selected by the algorithms, we refer to these methods as active matrix completion algorithms. The proposed techniques are generic and the same frameworks can be used in a wide variety of applications including recommendation systems, transductive / multi-label active learning, active learning in regression and active feature acquisition among others. Our extensive empirical analysis on several challenging real-world datasets certify the merit and versatility of the proposed frameworks in efficiently exploiting human intelligence in data mining / machine learning applications.

There is an increasing demand for applications that can detect changes in human affect or behavior especially in the fields of health care and crime detection. Detection of changes in continuous human affect dimensions from multimedia data precedes the exact prediction of an emotion as a continuum. With the growth in the dimensions of emotion space there is a need to discover latent descriptors (topics) that can explain these complex states. Considering that at every time step the audio/video frames constitute a set of such latent topics, the presence and absence of changes in emotion should effect the topics in those frames. Based on this assumption an Adaptive Temporal Topic model (ATTM) based change detection algorithm is presented that, at each time step, detects whether a significant change in human affect has occurred. ATTM is a probabilistic topic model that extends Latent Dirichlet Allocation model by incorporating the temporal dependencies between human audio/video `documents' and generates refined topics. The topics assigned to a document by ATTM are adapted to the presence or absence of a change in the affect dimension at that time step. ATTM along with different regression models has been tested on the multimodal Audio Visual Emotion Challenge (AVEC 2012) data and has shown promising results in comparison to existing temporal and non-temporal topic models.

Narrations of visual content, or audio descriptions, enhance the accessibility of video media for individuals who are blind. While the adoption of this assistive technology continues to broaden, audio descriptions have several limitations: they often lack detail (i.e., are abridged) to fit within dialogue-free gaps, and are difficult to use in fast-paced sequences or scenes of continuous dialogue. Haptic descriptions offer a promising augmentation to audio descriptions by conveying information through the sense of touch. This work presents audio-haptic descriptions with a user evaluation showing how haptic descriptions can add relevant information to films for improved comprehension.

Active learning and transfer learning are two different methodologies that address the common problem of insufficient labels. Transfer learning addresses this problem by using the knowledge gained from a related and already labeled data source, whereas active learning focuses on selecting a small set of informative samples for manual annotation. Recently, there has been much interest in developing frameworks that combine both transfer and active learning methodologies. A few such frameworks reported in literature perform transfer and active learning in two separate stages. In this work, we present an integrated framework that performs transfer and active learning simultaneously by solving a single convex optimization problem. The proposed framework computes the weights of source domain data and selects the samples from the target domain data simultaneously, by minimizing a common objective of reducing distribution difference between the data set consisting of reweighted source and the queried target domain data and the set of unlabeled target domain data. Comprehensive experiments on three real world data sets demonstrate that the proposed method improves the classification accuracy by 5% to 10% over the existing two-stage approach

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Multimedia applications like face recognition and facial expression recognition inherently rely on the availability of a large amount of labeled data to train a robust recognition system. In order to induce a reliable classification model for a multimedia pattern recognition application, the data is typically labeled by human experts based on some domain knowledge. However, manual annotation of a large number of images is an expensive process in terms of time, labor and human expertise. This has led to the development of active learning algorithms, which automatically identify the salient instances from a given set of unlabeled data and are effective in reducing the human annotation effort to train a classification model. Further, to address the possible presence of multiple labeling oracles, there have been efforts towards a batch form of active learning, where a set of unlabeled images are selected simultaneously for labeling instead of a single image at a time. Existing algorithms on batch mode active learning concentrate only on the development of a batch selection criterion and assume that the batch size (number of samples to be queried from an unlabeled set) to be specified in advance. However, in multimedia applications like face/facial expression recognition, it is difficult to decide on a batch size in advance because of the dynamic nature of video streams. Further, multimedia applications like facial expression recognition involve a fuzzy label space because of the imprecision and the vagueness in the class label boundaries. This necessitates a BMAL framework, for fuzzy label problems. To address these fundamental challenges, we propose two novel BMAL techniques in this work: (i) a framework for dynamic batch mode active learning, which adaptively selects the batch size and the specific instances to be queried based on the complexity of the data stream being analyzed and (ii) a BMAL algorithm for fuzzy label classification problems. To the best of our knowledge, this is the first attempt to develop such techniques in the active learning literature.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on varied criteria. In this paper we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimizes the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and a biomedical image dataset demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. Our empirical studies on UCI datasets show that incorporation of uncertainty information improves performance at later iterations while our studies on 20 Newsgroups dataset show that transfer learning improves the performance of the classifier during initial iterations.

Feedback related to motor performance is integral to improving the control, timing and coordination of movements. However, motor learning traditionally occurs within a group setting, limiting the quality of instruction and feedback. Even during one-on-one instruction, there are impediments to feedback such as physical separation between trainer and trainee, common in many sports such as snowboarding and swimming. We propose an inexpensive solution for real-time vibrotactile positioning and speed feedback that can complement traditional motor learning, and is compatible with existing vibrotactile motor instructions. We present a psychophysical study that examined participants' initial reactions to feedback stimuli pertaining to position and speed adjustments. Results support the proposed design in terms of both usability and naturalness, and provide insight into participants' conceptualization of feedback signals and feedback for rotational movements.

We consider the characterization of muscle fatigue through a noninvasive sensing mechanism such as Surface ElectroMyoGraphy (SEMG). While changes in the properties of SEMG signals with respect to muscle fatigue have been reported in the literature, the large variation in these signals across different individuals makes the task of modeling and classification of SEMG signals challenging. Indeed, the variation in SEMG parameters from subject to subject creates differences in the data distribution. In this article, we propose two transfer learning frameworks based on the multisource domain adaptation methodology for detecting different stages of fatigue using SEMG signals, that addresses the distribution differences. In the proposed frameworks, the SEMG data of a subject represent a domain; data from multiple subjects in the training set form the multiple source domains and the test subject data form the target domain. SEMG signals are predominantly different in conditional probability distribution across subjects. The key feature of the first framework is a novel weighting scheme that addresses the conditional probability distribution differences across multiple domains (subjects) and the key feature of the second framework is a two-stage domain adaptation methodology which combines weighted data from multiple sources based on marginal probability differences (first stage) as well as conditional probability differences (second stage), with the target domain data. The weights for minimizing the marginal probability differences are estimated independently, while the weights for minimizing conditional probability differences are computed simultaneously by exploiting the potential interaction among multiple sources. We also provide a theoretical analysis on the generalization performance of the proposed multisource domain adaptation formulation using the weighted Rademacher complexity measure. We have validated the proposed frameworks on Surface ElectroMyoGram signals collected from 8 people during a fatigue-causing repetitive gripping activity. Comprehensive experiments on the SEMG dataset demonstrate that the proposed method improves the classification accuracy by 20% to 30% over the cases without any domain adaptation method and by 13% to 30% over existing state-of-the-art domain adaptation methods.

The 14 papers in this special issue are extended versions of papers presented at ICME 2011, held in Barcelona, Spain, on 11-15 July 2011.

We propose a method for dynamic batch mode active learning where the batch size and selection criteria are integrated into a single formulation.

Active learning techniques have gained popularity to reduce human effort in labeling data instances for inducing a classifier. When faced with large amounts of unlabeled data, such algorithms automatically identify the exemplar and representative instances to be selected for manual annotation. More recently, there have been attempts towards a batch mode form of active learning, where a batch of data points is simultaneously selected from an unlabeled set. Real-world applications require adaptive approaches for batch selection in active learning. However, existing work in this field has primarily been heuristic and static. In this work, we propose a novel optimization-based framework for dynamic batch mode active learning, where the batch size as well as the selection criteria are combined in a single formulation. The solution procedure has the same computational complexity as existing state-of-the-art static batch mode active learning techniques. Our results on four challenging biometric datasets portray the efficacy of the proposed framework and also certify the potential of this approach in being used for real world biometric recognition applications.

We consider the characterization of muscle fatigue through noninvasive sensing mechanism such as surface electromyography (SEMG). While changes in the properties of SEMG signals with respect to muscle fatigue have been reported in the literature, the large variation in these signals across different individuals makes the task of modeling and classification of SEMG signals challenging. Indeed, the variation in SEMG parameters from subject to subject creates differences in the data distribution. In this paper, we propose a transfer learning framework based on the multi-source domain adaptation methodology for detecting different stages of fatigue using SEMG signals, that addresses the distribution differences. In the proposed framework, the SEMG data of a subject represent a domain; data from multiple subjects in the training set form the multiple source domains and the test subject data form the target domain. SEMG signals are predominantly different in conditional probability distribution across subjects. The key feature of the proposed framework is a novel weighting scheme that addresses the conditional probability distribution differences across multiple domains (subjects). We have validated the proposed framework on Surface Electromyogram signals collected from 8 people during a fatigue-causing repetitive gripping activity. Comprehensive experiments on the SEMG data set demonstrate that the proposed method improves the classification accuracy by 20% to 30% over the cases without any domain adaptation method and by 13% to 30% over the existing state-of-the-art domain adaptation methods.

Multi-label classification is a generalization of conventional classification, where it is possible for a single data point to have multiple labels. Manual annotation of a multi-label data point requires a human oracle to consider the presence/absence of every possible class separately, which involves significant labor. Active learning techniques are effective in reducing human labeling effort to induce a classification model. When exposed to large quantities of unlabeled data, such algorithms automatically select the salient and representative instances for manual annotation. Further, to address the high redundancy in data such as image or video sequences as well as the availability of multiple labeling agents, there have been recent attempts towards a batch mode form of active learning, where a batch of data points is selected simultaneously from an unlabeled set. In this work, we propose a novel optimization based batch mode active learning strategy to minimize human labeling effort in multi-label classification problems. To the best of our knowledge, this is the first attempt to develop such a scheme primarily intended for the multi-label context. The proposed framework is computationally simple, easy to implement and can be suitably modified to perform batch mode active learning in other formulations, such as single-label classification or problems involving hierarchical label spaces. Our results corroborate the efficacy of the proposed algorithm and certify the potential of the framework in being used for real world applications.

In this paper, we present a novel approach for teaching motor skills through the use of vibrotactile stimulation. We propose a kinematic-vibrotactile mapping that targets fundamental movements (basic building blocks of human motion) using saltatory vibration patterns where vibrations are delivered and interpreted as movement through a conceptual mapping. Two conceptual mappings are explored: the "follow me" concept and the push/pull metaphor. A user study, approved by a local ethics committee, was conducted to explore how these conceptual mappings affect learnability, recognition accuracy, response time and naturalness. Results show the approach to work effectively with a combination of vibration patterns under each conceptual mapping providing the most useful design.

With each technological breakthrough, our visual and auditory modalities are becoming increasingly overloaded with information. In this work, we explore the sense of touch as an alternative channel for communication for multimedia applications. We propose a novel theoretical framework, inspired by natural language, for the design of somatic information delivery systems, and validate this framework in applications of motor learning and accessible movies for individuals who are blind.

Discriminative learning when training and test data belong to different distributions is a challenging and complex task. Often times we have very few or no labeled data from the test or target distribution but may have plenty of labeled data from multiple related sources with different distributions. The difference in distributions may be in both marginal and conditional probabilities. Most of the existing domain adaptation work focuses on the marginal probability distribution difference between the domains, assuming that the conditional probabilities are similar. However in many real world applications, conditional probability distribution differences are as commonplace as marginal probability differences. In this paper we propose a two-stage domain adaptation methodology which combines weighted data from multiple sources based on marginal probability differences (first stage) as well as conditional probability differences (second stage), with the target domain data. The weights for minimizing the marginal probability differences are estimated independently, while the weights for minimizing conditional probability differences are computed simultaneously by exploiting the potential interaction among multiple sources. We also provide a theoretical analysis on the generalization performance of the proposed multi-source domain adaptation formulation using the weighted Rademacher complexity measure. Empirical comparisons with existing state-of-the-art domain adaptation methods using three real-world datasets demonstrate the effectiveness of the proposed approach.

SUMMARY: Images containing spatial expression patterns illuminate the roles of different genes during embryogenesis. In order to generate initial clues to regulatory interactions, biologists frequently need to know the set of genes expressed at the same time at specific locations in a developing embryo, as well as related research publications. However, text-based mining of image annotations and research articles cannot produce all relevant results, because the primary data are images that exist as graphical objects. We have developed a unique knowledge base (FlyExpress) to facilitate visual mining of images from Drosophila melanogaster embryogenesis. By clicking on specific locations in pictures of fly embryos from different stages of development and different visual projections, users can produce a list of genes and publications instantly. In FlyExpress, each queryable embryo picture is a heat-map that captures the expression patterns of more than 4500 genes and more than 2600 published articles. In addition, one can view spatial patterns for particular genes over time as well as find other genes with similar expression patterns at a given developmental stage. Therefore, FlyExpress is a unique tool for mining spatiotemporal expression patterns in a format readily accessible to the scientific community. AVAILABILITY: http://www.flyexpress.net CONTACT: [email protected].

Gesture spotting is the task of detecting and recognizing gestures defined in a vocabulary. The difficulty of gesture spotting stems from the fact that valid gestures appear sporadically in a continuous gesture stream, interspersed with invalid gestures (movements that do not correspond to any gesture contained in the vocabulary). In this paper, a novel method for designing threshold models from valid gesture models learnt through Adaptive Boosting is proposed. This threshold model is adaptive in nature and discriminates between valid and invalid gestures. Furthermore, a gesture spotting network consisting of the individual gesture models and the threshold model is proposed to perform the task of spotting and recognition simultaneously. This technique is evaluated in the context of spotting and recognizing activity gestures (hand gestures) from continuous accelerometer data streams. The proposed technique results in a precision of 0.78 and a recall of 0.93 out performing the HMM based threshold model which resulted in 0.4 and 0.81 precision and recall values.

The Conformal Predictions framework is a recent development in machine learning to associate reliable measures of confidence with results in classification and regression. This framework is founded on the principles of algorithmic randomness (Kolmogorov complexity), transductive inference and hypothesis testing. While the formulation of the framework guarantees validity, the efficiency of the framework depends greatly on the choice of the classifier and appropriate kernel functions or parameters. While this framework has extensive potential to be useful in several applications, the lack of efficiency can limit its usability. In this paper, we propose a novel kernel learning methodology to maximize efficiency in the CP framework. This method is validated using the k-Nearest Neighbors classifier on three different datasets, and our results show immense promise in applying this method to obtain efficient conformal predictors that can be practically useful.

Robust biometric recognition is of paramount importance in security and surveillance applications. In face based biometric systems, data is usually collected using a video camera with high frame rate and thus the captured data has high redundancy. Selecting the appropriate instances from this data to update a classification model, is a significant, yet valuable challenge. Active learning methods have gained popularity in identifying the salient and exemplar data instances from superfluous sets. Batch mode active learning schemes attempt to select a batch of samples simultaneously rather than updating the model after selecting every single data point. Existing work on batch mode active learning assume a fixed batch size, which is not a practical assumption in biometric recognition applications. In this paper, we propose a novel framework to dynamically select the batch size using clustering based unsupervised learning techniques. We also present a batch mode active learning strategy specially suited to handle the high redundancy in biometric datasets. The results obtained on the challenging VidTIMIT and MOBIO datasets corroborate the superiority of dynamic batch size selection over static batch size and also certify the potential of the proposed active learning scheme in being used for real world biometric recognition applications.

Cooking activities are complex activities consisting of multiple steps or tasks. These tasks can be associated with one another based on two properties the temporal structure that defines the sequence of occurrence of tasks and the objects that are used in the activity. This paper develops cooking activity models for the purpose of task prediction based on these two properties. The temporal structure of the sequence of tasks is captured by the novel hierarchical state space markov chain (HMC) and the object usage is represented using the object based task group (OTG) models. A probabilistic task prediction algorithm that fuses the HMC and OTG models has been developed to predict the next most probable task, given that a sequence of tasks has been completed. The proposed models and algorithms have been evaluated on two complex cooking activities making brownies and making eggs, achieving a subject independent accuracy of 68.5% for predicting tasks, which is an improvement by an average of 6% in comparison to a Markov chain. The work done in the paper is first of its kind as it focuses on task prediction rather than task recognition. The task prediction framework described in the paper can be easily adapted to any complex activity supporting various annotation schemes and activity models.

In this paper we present a new perspective into developing technologies for enriching social presence among remote interaction partners. Inspired by the abilities and limitations faced by people who are disabled during their everyday social interactions, we propose novel portable and wearable technologies that could potentially enrich remote interactions even in audio and video deprived settings. We describe the important challenges faced by people who are disabled during everyday dyadic and group social interactions and correlate them to the challenges faced by participants in remote interactions. With a case study of visually impaired individuals, we demonstrate how assistive technologies developed for social assistance of people who are disabled can help in increasing the social situational awareness, and hence social presence, of remote interaction partners.

Radiological images constitute a special class of images that are captured (or computed) for a specific purpose (i.e. diagnosis) and their ldquocorrectrdquo interpretation is vitally important. However, because they are not ldquonaturalrdquo images, radiologists must be trained to visually interpret them. This training involves perceptual learning that is gradually acquired over an extended period of exposure to radiological images. This implicit (subconscious) knowledge is difficult to pass along explicitly (i.e. verbally) to less experienced radiologists. Multimedia technology has the potential to facilitate perceptual learning in new radiologists. However, it is important to have an objective and quantitative method for evaluating the progress of trainees using this approach. This paper proposes an eye-tracker-based metric for determining the level of expertise of a radiologist in training, based on where he/she lies along a scale based on the visual scanning behavior of radiologists, ranging from novice to expert.

Drug Eluting Stents (DES) have distinct advantages over other Percutaneous Coronary Intervention procedures, but have recently been associated with the development of serious complications after the procedure. There is a growing need for understanding the risk of these complications, which has led to the development of simple statistical models. In this work, we have developed a predictive model based on Support Vector Machines on a real world live dataset consisting of clinical variables of patients being treated at a cardiac care facility to predict the risk of complications at 12 months following a DES procedure. A significant challenge in this work, common to most clinical machine learning datasets, was imbalanced data, and our results showed the effectiveness of the Synthetic Minority Over-sampling Technique (SMOTE) to address this issue. The developed predictive model provided an accuracy of 94% with a 0.97 AUC (Area under ROC curve), indicating high potential to be used as a decision support for management of patients following a DES procedure in real-world cardiac care facilities.

Processing of accelerometer data for recognizing short duration hand movements is a challenging problem. This paper focuses on characterization of acceleration data corresponding to hand movements (lift to mouth, scoop, stir, pour, unscrew cap) using aggregate statistical features and histograms computed from raw acceleration and derivative of the acceleration data. Data collected from an accelerometer placed on the wrist of subjects was used to perform the analysis. Supplementing the statistical features with raw acceleration histograms had a very marginal effect on the classification performance. However, the addition of derivative histograms resulted in a considerable improvement in the classification accuracy by nearly 8%. The effect of bin size of the derivative histograms was also conducted. It was observed that having a small number of bins decreased the classification accuracy by 3%. We thus show that adding features that capture the distribution of the changes in the acceleration data improve the classification performance.

Modern cell phones are becoming increasingly complex, as more and more features are added to every new generation. These complex user interfaces can make even placing a call difficult for those with disabilities. This presentation demonstrates a fully-functional version of PhotoTacs, an easy-to-use image-based phone book, designed specifically for those with cognitive disabilities and visual impairments.

The act of note-taking is a key component of learning in secondary and post-secondary classrooms. Students who take notes retain information from classroom lectures better, even if they never refer to those notes afterward. However, students who are legally blind, and who wish to take notes in their classrooms are at a disadvantage. Simply equipping classrooms with lecture recording systems does not substitute for note taking, since it does not actively engage the student in note-taking during the lecture. In this paper we detail the problems encountered by one math and computer science student who is legally blind, and we present our proposed solution: the CUbiC Note-Taker, which is a highly portable device that requires no prior classroom setup, and does not require lecturers to adapt their presentations. We also present results from two case studies of the Note-Taker, totaling more than 200 hours of in-class use.

Classroom note-taking has been shown to be beneficial, even if the student never reviews his/her own notes. Students that are legally blind are thus at a disadvantage because they face significant barriers to note-taking in the classroom. This presentation demonstrates a working prototype of the CUbiC Note Taker, which is a highly portable device that allows students who are legally blind to take their own notes in class without any special in-classroom accommodations, and without requiring lecturers to adapt their presentation in any way.

The advent of wearable sensors like accelerometers has opened a plethora of opportunities to recognize human activities from other low resolution sensory streams. In this paper we formulate recognizing activities from accelerometer data as a classification problem. In addition to the statistical and spectral features extracted from the acceleration data, we propose to extract features that characterize the variations in the first order derivative of the acceleration signal. We evaluate the performance of different state of the art discriminative classifiers like, boosted decision stumps (AdaBoost), support vector machines (SVM) and regularized logistic regression (RLogReg) under three different evaluation scenarios (namely subject independent, subject adaptive and subject dependent). We propose a novel computationally inexpensive methodology for incorporating smoothing classification temporally, that can be coupled with any classifier with minimal training for classifying continuous sequences. While a 3% increase in the classification accuracy was observed on adding the new features, the proposed technique for continuous recognition showed a 2.5 - 3% improvement in the performance.

The estimation of head pose angle from face images is an integral component of face recognition systems, human computer interfaces and other human-centered computing applications. To determine the head pose, face images with varying pose angles can be considered to be lying on a smooth low-dimensional manifold in high-dimensional feature space. While manifold learning techniques capture the geometrical relationship between data points in the high-dimensional image feature space, the pose label information of the training data samples are neglected in the computation of these embeddings. In this paper, we propose a novel supervised approach to manifold-based non-linear dimensionality reduction for head pose estimation. The Biased Manifold Embedding (BME) framework is pivoted on the ideology of using the pose angle information of the face images to compute a biased neighborhood of each point in the feature space, before determining the low-dimensional embedding. The proposed BME approach is formulated as an extensible framework, and validated with the Isomap, Locally Linear Embedding (LLE) and Laplacian Eigen-maps techniques. A Generalized Regression Neural Network (GRNN) is used to learn the non-linear mapping, and linear multi-variate regression is finally applied on the low-dimensional space to obtain the pose angle. We tested this approach on face images of 24 individuals with pose angles varying from -90deg to +90deg with a granularity of 2. The results showed substantial reduction in the error of pose angle estimation, and robustness to variations in feature spaces, dimensionality of embedding and other parameters.

The goal of image chromatic adaptation is to remove the effect of illumination and to obtain color data that reflects precisely the physical contents of the scene. We present in this paper an approach to image chromatic adaptation using Neural Networks (NN) with application for detecting--adapting human skin color. The NN is trained on randomly chosen color images containing human subject under various illuminating conditions, thereby enabling the model to dynamically adapt to the changing illumination conditions. The proposed network predicts directly the illuminant estimate in the image so as to adapt to human skin color. The comparison of our method with Gray World, White Patch and NN on White Patch methods for skin color stabilization is presented. The skin regions in the NN stabilized images are successfully detected using a computationally inexpensive thresholding operation. We also present results on detecting skin regions on a data set of test images. The results are promising and suggest a new approach for adapting human skin color using neural networks.

This demonstration will show the features and function of the portable iCARE Reader device, which allows people who are blind or visually impaired to read books (and other forms of printed text) in a more natural and convenient way than current tabletop flatbed scanner systems.

Color is an integral part of spatial perception and there is a need to develop systems that render color information accessible to blind individuals. A novel system that allows learning, presentation and analysis of color information, designed in consultations with focus groups of individuals who are blind is proposed. Our system is based on a methodology that renders colors as textures through a haptic device. The aim of the proposed approach is to enable color perception and provide a basis for assessing color similarity. Initial testing of the system shows that both blind individuals and sighted individuals can recognize colors through our approach and further assess similarity between colors through the system. A space was obtained through multidimensional scaling performed on similarity scores between pairs of colors as presented through our system. This space obtained high congruency with the chromaticity diagram and the hue saturation color wheel which shows the validity of our system to allow color visualization. A realtime system based on the proposed mapping is designed to allow realtime color perception.

Abstract — Video streaming over networks, such as wireless and Internet, is a challenging problem, due to network heterogeneity and variability of the visual content. Various scalable video coding techniques have been proposed to facilitate adaptable transmission of video streams. In this paper, we compare the bandwidth utilization of such scalable schemes for video streaming to receivers of diverse capacities. The results reveal the superiority of scalable schemes that adopt bit plane coding, such as Fine Granularity Scalability (FGS) and Progressive-FGS (PFGS). In addition, we evaluate the reconstructed qualities of FGS and PFGS streams due to heterogeneity of receiver capacities and variability of the transmission channel. The results suggest that, for low bit rate receivers/channels, FGS streams can be used instead of PFGS streams, thus reducing the computational overhead at the video receiver.

In this paper, we present an image-text dewarping methodology based on robust estimation of text-lines. When a text-page is captured by a camera, it suffers both from the perspective distortions and the page curl. The non-linear distortion due to page-curl is inherently present, given the surface nature of the pages and the text-book. The state-of-the-art OCR systems have a very low performance on recognizing such distorted text. To remove both these distortions and to produce a flattened view of the text, we use the cues present in the image-text, i.e., the text-lines on the surface of the page are straight. The methodology requires only a single camera captured image and does not require any calibration or any other expensive hardware setups as in other methods. Experimental results on a set of documents show that the methodology produces visually pleasing output and also improves OCR accuracy

Surgical movement is composed of discrete gestures that are combined to perform complex surgical procedures. A promising approach to objective surgical skill evaluation systems is kinematics and kinetic analysis of hand movement that yields a gesture level analysis of proficiency of a performed movement. In this paper, we propose a novel system that combines surgical gesture segmentation, surgical gesture recognition, and expertise analysis of surgical profiles in minimally invasive surgery (MIS). Kinematic analysis was used to segment gestures from a continuous motion stream. Human anatomy driven Hidden Markov Models (HMMs) are adopted for gesture recognition and expertise identification. When the proposed system was tested on a library of 200 samples for every basic surgical gesture, the gesture recognition module reported a perfect accuracy rate for the basic gestures, while the expertise identification module showed 94.7% accuracy.

A variety of error resilience and scalable coding techniques have recently been proposed to facilitate the delivery of video over best-effort networks; a common drawback of these techniques is reduced compression efficiency. Also, MPEG-7 descriptors have recently been developed for the purpose of indexing. In this paper, we propose to employ MPEG-7 descriptors to improve the quality of the video delivered over best-effort networks. In particular, we propose a video transmission system that uses the motion activity descriptors to ensure robust video transmission. A novel motion activity extraction technique is proposed, which relies on a neural network approach. By considering several low-level visual features, our proposed extraction approach achieves high consistency with subjective evaluations of motion activities. In order to demonstrate the benefits of the proposed transmission system, we develop a selective packet dropping scheme that can be applied in case of network congestion. Simulations demonstrate that the reconstruction quality of the proposed congestion scheme can surpass conventional schemes by 1.2 dB. The network performance of the proposed transmission system when video sequences are coded into single layer or scalable layers is presented. We also present a transcoding scheme that achieves the optimal reconstructed quality by exploiting the motion activities of the underlying video sequence

Today's multimedia applications demand sophisticated compression and classification techniques in order to store, transmit, and retrieve audio-visual information efficiently. Over the last decade, perceptually based image compression methods have been gaining importance. These methods take into account the abilities (and the limitations) of human visual perception (HVP) when performing compression. The upcoming MPEG 7 standard also addresses the need for succinct classification and indexing of visual content for efficient retrieval. However, there has been no research that has attempted to exploit the characteristics of the human visual system to perform both compression and classification jointly. One area of HVP that has unexplored potential for joint compression and classification is spatial frequency perception. Spatial frequency content that is perceived by humans can be characterized in terms of three parameters, which are: 1) magnitude; 2) phase; and 3) orientation. While the magnitude of spatial frequency content has been exploited in several existing image compression techniques, the novel contribution of this paper is its focus on the use of phase coherence for joint compression and classification in the wavelet domain. Specifically, this paper describes a human visual system-based method for measuring the degree to which an image contains coherent (perceptible) phase information, and then exploits that information to provide joint compression and classification. Simulation results that demonstrate the efficiency of this method are presented.

Haptic perception refers to the ability of human beings to perceive spatial properties through touch-based sensations. In haptics, contextual clues about material,shape, size, texture, and weight configurations of an object are perceived by individuals leading to recognition of the object and its spatial features. In this paper, we present strategies and algorithms to model context in haptic applications that allow users to haptically explore objects in virtual reality/augmented reality environments. Initial results show significant improvement in accuracy and efficiency of haptic perception in augmented reality environments when compared to conventional approaches that do not model context in haptic rendering.

This paper describes the iCare Interaction Assistant, an assistive device for helping the individuals who are visually impaired during social interactions. The research presented here addresses the problems encountered in implementing real-time face recognition algorithms on a wearable device. Face recognition is the initial step towards building a comprehensive social interaction assistant that will identify and interpret facial expressions, emotions and gestures. Experiments conducted for selecting a face recognition algorithm that works despite changes in facial pose and illumination angle are reported. Performance details of the face recognition algorithms tested on the device are presented along with the overall performance of the system. The specifics of the hardware components used in the wearable device are mentioned and the block diagram of the wearable system is explained in detail.

This presentation demonstrates a working prototype of the iCare Interaction Assistant, a wearable assistive device based on research aimed at facilitating the social interactions of people who are blind or visually impaired. Using a tiny unobtrusive camera mounted inside the nose bridge of a pair of eyeglasses, this prototype is able to learn and recognize faces at a distances up to 10 feet, thus allowing the user to initiate conversations with persons in their vicinity, without waiting for others to approach them. Ongoing work is aimed at facilitating the subsequent verbal interaction by recognizing and interpreting non-verbal communication, including eye contact, facial expressions, emotions, and gestures.

In this paper, we present a methodology for precisely comparing the robustness of face recognition algorithms with respect to changes in pose angle and illumination angle. For this study, we have chosen four widely-used algorithms: two subspace analysis methods (principle component analysis (PCA) and linear discriminant analysis (LDA)) and two probabilistic learning methods (hidden Markov models (HMM) and Bayesian intra-personal classifier (BIC)). We compare the recognition robustness of these algorithms using a novel database (FacePix) that captures face images with a wide range of pose angles and illumination angles. We propose a method for deriving a robustness measure for each of these algorithms, with respect to pose and illumination angle changes. The results of this comparison indicate that the subspace methods perform more robustly than the probabilistic learning methods in the presence of pose and illumination angle changes.

This paper proposes a two-step methodology for improving the discriminatory power of linear discriminant analysis (LDA) for video-based human face recognition. Results indicate that, under real-world video capture conditions, face images extracted from a video sequence have enough 3D rotations, illumination changes and background variations to reduce the discriminatory power of an LDA classifier. The proposed method involves deriving an LDA subspace from carefully selected subsets of face images that fall within a narrow range of pose angles, and then growing the classification regions in the LDA subspace using face images with a wider range of pose angle changes, illumination changes, and background variations. Polynomial support vector machines (SVM) are shown to provide better recognition rates by defining the boundaries between clusters that represent the faces of different subjects. Results show that there is an improvement in the recognition rate when the LDA subspace is derived with this methodology than when it is derived with a set of face images with a widely divergent set pose angles, illumination variations, and backgrounds.

The goal of image chromatic adaptation is to remove the effect of illumination and to obtain color data that reflects precisely the physical contents of the scene. We present An approach to image chromatic adaptation using neural networks (NN) with application for detecting - adapting human skin color. The network is trained on randomly chosen color images containing human subject under various illuminating conditions, thereby enabling the model to dynamically adapt to the changing illumination conditions. The proposed network predicts directly the illuminant estimate in the image so as to adapt to the human skin color. The comparison of our method with gray world, white patch and neural network on white patch algorithms is presented. We also present our results on detecting skin regions in NN color corrected test images. The results are promising and suggest a new approach for adapting human skin color using NN's.

BACKGROUND: Modern developmental biology relies heavily on the analysis of embryonic gene expression patterns. Investigators manually inspect hundreds or thousands of expression patterns to identify those that are spatially similar and to ultimately infer potential gene interactions. However, the rapid accumulation of gene expression pattern data over the last two decades, facilitated by high-throughput techniques, has produced a need for the development of efficient approaches for direct comparison of images, rather than their textual descriptions, to identify spatially similar expression patterns. RESULTS: The effectiveness of the Binary Feature Vector (BFV) and Invariant Moment Vector (IMV) based digital representations of the gene expression patterns in finding biologically meaningful patterns was compared for a small (226 images) and a large (1819 images) dataset. For each dataset, an ordered list of images, with respect to a query image, was generated to identify overlapping and similar gene expression patterns, in a manner comparable to what a developmental biologist might do. The results showed that the BFV representation consistently outperforms the IMV representation in finding biologically meaningful matches when spatial overlap of the gene expression pattern and the genes involved are considered. Furthermore, we explored the value of conducting image-content based searches in a dataset where individual expression components (or domains) of multi-domain expression patterns were also included separately. We found that this technique improves performance of both IMV and BFV based searches. CONCLUSIONS: We conclude that the BFV representation consistently produces a more extensive and better list of biologically useful patterns than the IMV representation. The high quality of results obtained scales well as the search database becomes larger, which encourages efforts to build automated image query and retrieval systems for spatial gene expression patterns.

The International Standards Organization (ISO) has proposed a family of standards for compression of image and video sequences, including the JPEG, MPEG-1, and MPEG-2. The latest MPEG-4 standard has many new dimensions to coding and manipulation of visual content. A video sequence usually contains a background object and many foreground objects. Portions of this background may not be visible in certain frames due to the occlusion of the foreground objects or camera motion. MPEG-4 introduces the novel concepts of video object planes (VOPs) and Sprites. A VOP is a visual representation of real world objects with shapes that need not be rectangular. Sprite is a large image composed of pixels belonging to a video object visible throughout a video segment. Since a sprite contains all parts of the background that were at least visible once, it can be used for direct reconstruction of the background VOP. Sprite reconstruction is dependent on the mode in which it is transmitted. In the static sprite mode, the entire sprite is decoded as an Intra VOP before decoding the individual VOPs. Since sprites consist of the information needed to display multiple frames of a video sequence, they are typically much larger than a single frame of video. Therefore, a static sprite can be considered as a large static image. In this paper, a novel solution to address the problem of spatial scalability has been proposed, where the sprite is encoded in discrete wavelet transform (DWT). A lifting kernel method of DWT implementation has been used for encoding and decoding sprites. Modifying the existing lifting scheme while maintaining it to be shape-adaptive results in a reduced complexity. The proposed scheme has the advantages of: 1) avoiding the need for any extensions to image or tile border pixels and is hence superior to the discrete cosine transform-based low latency scheme (used in the current MPEG-4 verification model) and 2) mapping the in place computed wavelet coefficients into a zero-tree structure without actually rearranging them, thereby saving allocation of additional memory. The proposed solutions provide efficient implementation of the sprite decoder, making possible a VLSI realization with a reduced real estate.

Transmission of compressed video over unreliable networks such as the Internet will inevitably result in serious degradations in video quality. Therefore, multiple description coding (MDC) encodes the video signal into a number of bit streams (descriptions) such that receiving a description improves the video quality. Among the different MDC schemes, spatial/temporal sub-sampling is a low complexity encoding/decoding scheme, however these MDC schemes are vulnerable to varying channel conditions and visual content. To guarantee better image quality during transmission, we propose an adaptive scheme that alternates between two simple MDC schemes depending on the channel status and underlying visual content. The proposed scheme achieves substantial robustness over packet lossy networks. Furthermore, a multilayer perceptron (MLP) network is designed to provide an on-line decision mechanism with 85% accuracy. Experimental results suggest that our approach can achieve a superior video quality for a wide range of packet loss and video content.

Complex human motion sequences (such as dances) are typically analyzed by segmenting them into shorter motion sequences, called gestures. However, this segmentation process is subjective, and varies considerably from one human observer to another. In this paper, we propose an algorithm called hierarchical activity segmentation. This algorithm employs a dynamic hierarchical layered structure to represent the human anatomy, and uses low-level motion parameters to characterize motion in the various layers of this hierarchy, which correspond to different segments of the human body. This characterization is used with a naive Bayesian classifier to derive creator profiles from empirical data. Then those profiles are used to predict how creators will segment gestures in other motion sequences. When the predictions were tested with a library of 3D motion capture sequences, which were segmented by 2 choreographers they were found to be reasonably accurate.

There are approximately 10 million blind and visually-impaired persons in the United States. As the information economy accounts for a larger percent of jobs, people who are blind face barriers when competing for jobs that require vision to access, absorb, and use information. While much research effort has been expended toward developing assistive devices for the blind, many of the technological solutions proposed are unworkable. Many of these assistive devices fall short of meeting the needs of users, and sometimes even create more problems than they solve. In order to avoid repeating these mistakes, a user-centric approach is vital to the design of assistive devices. This paper describes our own efforts to develop and employ such a design methodology in the iCare project. In this project, the research efforts are being focused on those problems that will have the greatest impact on the day-to-day life of the user, and that are technologically feasible.

Wavelets are used in most of the recent image compression standards, because they provide scalability, multi-resolution, and energy concentration. Most of these wavelet image coders employ bi-orthogonal filters. This provides the advantages of selective frequency localization and Perfect Reconstruction (PR), compared to classical orthogonal filters. Bi-orthogonal filters also obviate the need for any phase compensation in the pyramidal filter structure. Lifting has emerged as a powerful method for implementing bi-orthogonal wavelet filters. It exploits the similarity of the filter coefficients between the low pass and high pass filters to provide a higher speed of execution, compared to conventional convolution-based coders. In this paper, we propose a novel lifting-based system that achieves better compression than existing wavelet kernels. The proposed lifting-based system uses bi-orthogonal filters in its architecture, and can operate in the JPEG2000 framework. In the proposed system, the coefficients of either the low-pass or high-pass filter can be modified, without changing the coefficients of the other filter. This is done in a manner that enhances compression while preserving the perfect reconstruction (PR) condition, and increasing the regularity of the modified pair of filters. Extensive simulations have been performed in the JPEG2000 domain to compare the efficiency of our kernels to the lifting wavelet kernels provided by the JPEG2000 standard

Multimedia processing is becoming increasingly important with a wide variety of applications ranging from multimedia cellphones to high-definition interactive television. Media processing involves the capture, storage, manipulation and transmission of multimedia objects such as text, handwritten data, audio objects, still images, 2D/3D graphics, animation, and full-motion video. A number of implementation strategies have been proposed for processing multimedia data. These approaches can be broadly classified based on the evolution of processing architectures and the functionality of the processors. In order to provide media processing solutions to different consumer markets, designers have combined some of the classical features from both the functional and evolution-based classifications resulting in many hybrid solutions. We propose a categorization of existing microprocessors based on a combination of both architectural and functional flavors with examples of each approach from the latest multimedia processing families. The varying processing requirements in multimedia computing for reconfigurable multimedia processing are presented.

The recent MPEG 4 and JPEG 2000 standards address the need for content based coding and manipulation of visual media. The upcoming MPEG 7 standard proposes content descriptors, which succinctly describe the visual content for the purpose of efficient retrieval. This implies that there is an impending need for efficient and effective joint compression and indexing approaches. Several compressed domain indexing techniques have been presented in the recent literature. These are based on the extraction of features from the compression parameters to derive the indices. However, there is little work in the domain of exploring the use of these features to serve the purposes of both compression and indexing. In this paper, we propose a new technique for joint compression and indexing in the wavelet domain. We note that wavelet based compression is used in JPEG 2000 and (for texture coding) in MPEG 4. In the proposed technique, the wavelet decomposed image is first preprocessed to extract features which are then used for compressing the image as well as for deriving the indices. Extensive simulations have been performed in the JPEG 2000 compressed domain to demonstrate the efficiency of the proposed technique.

In this paper, we present an efficient term rewriting technique that computes a degree of term to domain relevance. The proposed method resolves the problems in ontology integrated concept search. Those problems are (i) Pre-defined concept classes in ontology are not relevant to users (no proper concept class for a target annotation has not found). (ii) Too many similar concept classes are provided to a user therefore, a user may fail to choose a correct semantic class for a target annotation (ordinary users are not an expert in concept classification). The method uses sense disambiguation task for finding relevant terms for a given domain. Sense disambiguation requires term-to-term similarity measurement and term frequency measurement. For fair modeling of not observed term frequencies, discounting and redistribution model is applied. The proposed method is a compliment to our previous work presented in [13][14]. Robustness of our method is demonstrated through human judgment test that shows our method allows prediction of precise term list (overall 75% of correct prediction) that are relevant to a given domain.

Noting the shortage of IT professionals nationally [1], we propose a comprehensive curriculum that supports a variety of programs geared to all ages from early school years to retirement and beyond. Current IT workforce development efforts are limited to training, and have not as yet focused on education and professional development. Largely, this is due to a lack of a science underpinning for IT related curricula. Without such a unified science component, a structured organization of information related concepts cannot be derived.Our proposal includes the development of a number of programs addressing the needs of a variety of learners ranging from elementary school through college and beyond. Seven programs, each with a specific emphasis for various groups, are being developed. Such essential issues as industrial-academic liaisons, workforce (re)training, promotional and awareness programs, teacher training, and IT professional role redefinition, are integral pieces of this project. All developments will be firmly founded on the scientific framework of information science and engineering [2].This work is supported by NSF grant DUE-9950168.

Human color perception is subjective. In addition to RGB or HSV values representing color content, psychological factors, circumstantial factors, environmental factors, and physiological factors play an important role in encapsulating the color content. Typically, only the RGB or HSV values are used in indexing and retrieval of color images. In this paper, we demonstrate the superior retrieval performance of techniques which employ all of the above factors in color retrieval. We first present the variety of factors involved in human color perception and also an evaluation of the existing color indexing methods. Several interesting problems including comparison of images in the color-perceptual domain and retrieval by color affection are illustrated as potential novel image query mechanisms.

Digital video databases in compressed form are becoming widely available. In applications such as video browsing, and picture in picture, for a lower bitrate, there is a need to down-sample the video before transmission. The conventional approach to downscale a compressed video sequence is to decompress it at the video server, perform the down-sampling in the pixel domain and then recompress it for efficient delivery. This process is computationally intensive due to the motion estimation process required during the recompression phase. In the alternative compressed domain approach, the motion vectors of the downscaled video sequence are computed directly from the motion vectors of the original full size stream. In this paper we propose a compressed domain technique that generates a better estimate for the downscaled motion vectors. Simulations suggest that the performance achieved with the proposed method is superior by up to 1 dB to the current compressed domain techniques.

The advances in VLSI technology, high-speed processor designs, Internet/Intranet implementations, broadband networks (ATM and ISDN), and compression standards (JPEG, MPEG, H.261, H.263, and G.273) are leading to the popularity of multimedia applications. Examples include video over the Internet, interactive TV, distance learning, telemedicine, and digital libraries. Visual media processing poses challenges from several perspectives, specifically from the points of view of real-time implementation and scalability. We first present an overview and categorization of the various architectural approaches for multimedia processing. The fundamental operations involved in a majority of visual processing tasks are then derived. We propose an affine transform-processor-core-based video processor architecture called the fractal engine that is capable of implementing the basic visual processing operations. The fractal engine is an open architecture, and is designed to be modular and scalable, and therefore has the potential to satisfy the heterogeneous computing requirements of the different media types in multimedia processing. Details of the individual modules of the fractal engine as well the implementation of the architecture in VHDL are also presented.

Visual (image and video) database systems require efficient indexing to enable fast access to the images in a database. In addition, the large memory capacity and channel bandwidth requirements for the storage and transmission of visual data necessitate the use of compression techniques. Vector quantization (VQ) is an efficient technique for low bit-rate image and video compression. In addition, the low complexity of the decoder makes VQ attractive for low power systems and applications which require fast decoding. In this paper, we present an indexing technique for compressed video using vector quantization. Here, a video sequence is first compressed using VQ. Each frame is represented by a usage map, a set of VQ labels, and a set of motion vectors. The video sequence is partitioned into shots and the various camera operations and motion within each shot are then determined by processing the VQ label maps. Each shot is indexed using a spatio-temporal index. The spatial index refers to the spatial content of the representative frame of a shot, while the temporal index represents the temporal content of the shot. The spatial index is based on the codewords used to compress the representative frame, while the temporal index is based on motion and camera operations within the shot. The proposed indexing technique is executed entirely in the compressed domain. This entails significant savings in computational and storage costs resulting in faster execution.

In this paper, we discuss the design aspects of a dynamic distributed directory scheme (DDS) to facilitate efficient and transparent access to information files in mobile environments. The proposed directory interface enables users of mobile computers to view a distributed file system on a network of computers as a globally shared file system. In order to counter some of the limitations of wireless communications, we propose improvised invalidation schemes that avoid false sharing and ensure uninterrupted usage under disconnected and low bandwidth conditions.

We propose a reduced complexity wavelet-based image coding technique. Here, 64-D (for three stages of decomposition) vectors are formed by combining appropriate coefficients from the wavelet subimages, 16-D feature vectors are then extracted from the 64-D vectors on which vector quantization (VQ) is performed. At the decoder, 64-D vectors are reconstructed using a nonlinear interpolative technique. The proposed technique has a reduced complexity and has the potential to provide a superior coding performance when the codebook is generated using the training vectors drawn from similar images.

This paper presents a VLSI implementation of discrete wavelet transform (DWT). The architecture is simple, modular, and cascadable for computation of one or multidimensional DWT. It comprises of four basic units: input delay, filter, register bank, and control unit. The proposed architecture is systolic in nature and performs both high- and low-pass coefficient calculations with only one set of multipliers. In addition, it requires a small on-chip interface circuitry for interconnection to a standard communication bus. A detailed analysis of the effect of finite precision of data and wavelet filter coefficients on the accuracy of the DWT coefficients is presented. The architecture has been simulated in VLSI and has a hardware utilization efficiency of 87.5%. Being systolic in nature, the architecture can compute DWT at a data rate of N/spl times/10/sup 6/ samples/s corresponding to a clock speed of N MHz.

We propose a combined systolic array-content addressable memory architecture for real-time Gabor decomposition. We then present codec designs for progressive image transmission using this architecture. Gabor decomposition is attractive for image compression since the basis functions match the human visual profiles. Gabor functions also achieve the lowest bound on the joint uncertainty of data. However these functions are not orthogonal and hence an analytic solution for tire decomposition does not exist. It has been shown that Gabor decomposition can be computed as a multiplication between a transform matrix and a vector of the image data. For an n/spl times/n image, the proposed architecture for Gabor decomposition consists of a linear systolic array of n processing elements each with a local CAM. Simulations and complexity studies show that this architecture can achieve real-time performance with current VLSI technology.>

In this paper, we propose a unified associative memory architecture for real-time implementation of motion estimation and frame adaptive vector quantization for video compression. The proposed architecture has the advantages of simplicity, partitionability, and modularity and has hence the potential for VLSI implementation.>

The routing table lookup function is described, and recent research is reviewed, focusing on multiprocessor implementations. An implementation of the PATRICIA (practical algorithm to retrieve information coded in alphanumeric) radix searching algorithm on a linear pipelined array of processors is proposed. The implementation is described in detail and then evaluated in terms of the technology requirements, system performance, and robustness.>

In this paper, we propose an affine transform based vector quantization (ATVQ) technique for image coding applications. Vector quantization (VQ) is intrinsically superior to predictive coding, transform coding, and other suboptimal and ad hoc procedures. The limitation of VQ is the very large codebook that must be generated and stored. The proposed affine transform based vector quantization technique addresses this problem. The image to be coded is partitioned into disjoint square blocks. Each block is regarded as a vector and is encoded by searching through a set of affine transforms and a codebook of templates. The transform- template pair that can reconstruct an approximate input vector with minimum distortion is selected. The parameters and the index of the affine transform and the index of the template constitute the codeword of the input vector. In decoding, the image vector is reconstructed by applying the inverse of the affine transform on the template. ATVQ can reconstruct more input vectors without any distortion than conventional VQ can reconstruct, using the same codebook. Simulation results show that the technique performs well using a universal codebook. This technique is also suitable for progressive image transmission as its performance is good at very low bit rates.

In this paper, we propose an MPEG-like video coder using a 3-dimensional (3D) adaptive pyramid. In the 3D adaptive pyramid, the temporal or spatial contraction is selected using the corresponding prediction differences. A video codec is designed using the 3D adaptive pyramid and intra-frame vector quantization. Encoding errors introduced at the upper levels are fed-forward to the lower levels of the pyramid which makes lossless coding possible. The proposed pyramidal video coder has a number of similarities with the MPEG-I video compression standard. Simulation results for the CCITT standard test sequences indicate that the 3D adaptive pyramid reduces the lossless bit rate by a factor of two. Excellent subjective quality as well as objective quality (PSNR value of 36.6 db) are obtained at a bit rate less than T1 rate (i.e., 1.544 Mb/s). Furthermore, smooth transition is achieved in the case of scene changes without sacrificing picture quality.

Vector quantization (VQ) is a promising technique for low bit rate image coding. Recently image sequence coding algorithms based on VQ have been reported in the literature. We note that image sequences are highly non stationary and generally exhibit variations from frame to frame and from scene to scene; hence using a fixed VQ codebook to encode the different frames/sequences cannot guarantee a good coding performance. Several adaptive techniques which improve the coding performance have been reported. However, we note that most adaptive techniques result in further increases in the computational complexity and/or the bit rate. In this paper, a new frame adaptive VQ technique for image sequence coding (SC- FAVQ) is presented. This technique exploits the inter/intraframe correlations and provides frame adaptability at a reduced complexity. In addition, a dynamic self organized codebook is used to track the local statistics from frame to frame. Computer simulations using standard CCITT image sequences demonstrate the superior coding performance of SC-FAVQ.

Discrete wavelet transform is an ideal tool for multi-resolution representation of image signals. Some promising results have been recently reported on the application of wavelet transform for image compression. In this paper, we propose a new wavelet coding technique for image compression. The proposed scheme has the advantages of improved coding performance and reduced computational complexity. The input image is first decomposed into a pyramid structure with three layers using a 2-D wavelet transform. A block size of 2m - 3 (m equals 1, 2, 3) is used for each orientation sub-image at the m-th layer to form 64-D vectors by combining the corresponding blocks in all the sub-images. The 64-D vectors are then encoded using 16-D non-linear interpolative vector quantization (NIVQ). At the decoder, the indices are used to reconstruct the 64-D vectors directly from a 64-D codebook designed using a non-linear interpolative technique. The proposed scheme not only exploits the correlation among the wavelet sub-images but also preserves the high frequency sub-images. Simulation results show that the reconstructed image of a superior quality can be obtained at a compression ratio of about 100:1.

The authors propose a combined source-channel coding scheme which provides error protection in order to improve the coding performance over noisy channels for three frame replenishment coding techniques. The scheme provides a trade-off between the reconstruction accuracy associated with the source coder and the degree of error protection. The simulation results demonstrate that a small amount or error protection provides a significant improvement in performance without sacrificing transmission bandwidth.>

A systolic array architecture for image coding using adaptive vector quantization is presented. A basic systolic cell was designed with two modes of operation. In the forward mode, the cell executes the basic distortion operation, while in the reverse mode, the cell executes the centroid computation operation. Both modes co-exist in perfect synchronism. The systolic array module essentially consists of an array of L*N basic systolic cells connected in parallel and pipeline in the direction of the vector dimension, L, and codeword dimension, N, respectively. This architecture results in a speedup proportional to NL, and has the following advantages: there is no need for separate hardware to compute the new centroids; there is no need for a high speed interface to transfer the new centroids into the systolic array; and there are no delays involved in the computation and transfer of new centroids.>