Should organisations force business units to use shared IT resources, or should business units, notwithstanding the potential for duplication and higher costs, be allowed to select their own IT resources? If successful, efforts to expand IT resource relatedness will increase the sharing of IT resources across business units leading to higher business unit agility. Alternatively, forced IT resource sharing could remove IT optionality, thereby impeding business unit operations and agility. To investigate these opposing viewpoints, we consider how systems of complements involving three distinct IT resource classes (IT infrastructure, applications, and data) affect business unit agility. Which IT resources are shared matters as the complementarity effects from sharing various IT resources may be different from the marginal effects of sharing just one type of IT resource. We investigate these arguments using data from 120 organisations. Analysis shows that sharing applications and data enables business unit agility, while sharing IT infrastructure hurts business unit agility. Follow-up qualitative interviews with senior IT executives support and extend these findings. In combination, the results of our quantitative and qualitative studies enhance our understanding of how IT resource relatedness influences business unit agility, contingent on both the level and type of IT resource sharing.

Motor imagery (MI)-based brain computer interfaces (BCIs) frequently use convolutional neural networks (CNNs) to analyse electroencephalography (EEG) signals. In this study, we proposed a novel methodology that includes an innovative preprocessing step and a new model for MI EEG classification. In the preprocessing, we use common average reference (CAR) filtering and Laplace filtering of EEG signals. The CAR filter eliminates the overall noise and Laplace filter removes the neighbouring electrode noise. Additionally, a sliding window method is used to increases the number of small-time segments which prevents overfitting. Next, the time segments are converted into spectrograms using the short-time Fourier transform (STFT). Further, the concatenated spectrogram images of mu and beta bands are processed using a CNN model with self-attention. The proposed model uses both local and global information to effectively extract features The EEG signals obtained from BCI competition IV dataset-2a are divided into 80:20 ratio for training and testing. Moreover, the ablation study highlights the importance of the combination of CAR and Laplace filters. The classification results obtained using proposed methodology shows advancement as compared to state-of-the-art methods. Finally, the proposed CNN model learning and feature distribution are visualized with the gradient weight class activation map.

Agility has long been recognised as a key determinant of organisational performance. The pursuit of agility within business units (BUs) is challenging as IT executives in multi-business organisations (MBOs) are tasked with balancing a need for organisation-wide IT synergies against a desire among BUs for local IT solutions that enable agility in their end markets. MBOs are increasingly turning to corporate IT platforms to respond to their BUs’ IT needs but doing so presents a challenge to BU IT autonomy and, in turn, to BU agility. In this paper, we posit that BU agility is determined by resource synergies based on the interplay between corporate IT platforms and local, BU-controlled, IT applications. These applications lead to greater BU process digitisation and, in turn, to expanded BU agility. We evaluate our model using data from an international survey of IT executives in MBOs. Our findings provide support for our model and, in so doing, identify different paths that MBOs can take to enhance BU agility. Our contribution affirms the value of corporate IT platforms for BU agility but suggests that MBOs should tread carefully when developing corporate IT platforms that coincide with mandatory use of shared IT applications.

The U.S. Defense Advanced Research Projects Agency's (DARPA) Neovision2 program aims to develop artificial vision systems based on the design principles employed by mammalian vision systems. Three such algorithms are briefly described in this paper. These neuromorphic-vision systems' performance in detecting objects in video was measured using a set of annotated clips. This paper describes the results of these evaluations including the data domains, metrics, methodologies, performance over a range of operating points and a comparison with computer vision based baseline algorithms.

Much attention is currently being paid in both the academic and practitioner literatures to the value that organisations could create through the use of big data and business analytics (Gillon et a...

Sharma and Yetton, in 2003, hypothesized and found empirical support for the moderating effect of task interdependence on the relationship between management support and implementation success. In this issue of EJIS, Hwang and Schmidt critique Sharma and Yetton's meta-analysis protocol, arguing that a random-effects model should have been employed, rather than the fixed-effect model they used. Further, Hwang and Schmidt claim that the moderating effect hypothesized by Sharma and Yetton is not supported when a random-effects model is employed. We re-analyze Sharma and Yetton's data by employing a dedicated, commercially available package for conducting random-effects meta-analysis. Our analysis finds no support for Hwang and Schmidt's claim. The random-effects meta-analysis supports Sharma and Yetton's original findings: task interdependence moderates the effect of management support on implementation success.

An integral part of interactive computing environments are systems that have the ability to process information about their users in real-time. In many cases it is desirable to not only recognize a human user but also to extract as much information about the user as possible, such as gender, ethnicity, age, etc. In this paper we present an FPGA implementation of a neural network configured specifically for performing face detection and gender classification in real-time video streams. Our streaming architecture performs the face and gender classification tasks at 30 frames per second on a small sized Virtex-4 FPGA, at accuracy comparable to that of a leading commercial software implementation.

This paper presents a spatio-temporal approach in recognizing six universal facial expressions from visual data and using them to compute levels of interest. The classification approach relies on a two-step strategy on the top of projected facial motion vectors obtained from video sequences of facial expressions. First a linear classification bank was applied on projected optical flow vectors and decisions made by the linear classifiers were coalesced to produce a characteristic signature for each universal facial expression. The signatures thus computed from the training data set were used to train discrete hidden Markov models (HMMs) to learn the underlying model for each facial expression. The performances of the proposed facial expressions recognition were computed using five fold cross-validation on Cohn-Kanade facial expressions database consisting of 488 video sequences that includes 97 subjects. The proposed approach achieved an average recognition rate of 90.9% on Cohn-Kanade facial expressions database. Recognized facial expressions were mapped to levels of interest using the affect space and the intensity of motion around apex frame. Computed level of interest was subjectively analyzed and was found to be consistent with "ground truth" information in most of the cases. To further illustrate the efficacy of the proposed approach, and also to better understand the effects of a number of factors that are detrimental to the facial expression recognition, a number of experiments were conducted. The first empirical analysis was conducted on a database consisting of 108 facial expressions collected from TV broadcasts and labeled by human coders for subsequent analysis. The second experiment (emotion elicitation) was conducted on facial expressions obtained from 21 subjects by showing the subjects six different movies clips chosen in a manner to arouse spontaneous emotional reactions that would produce natural facial expressions.

Current computing systems do not support human work effectively. They restrict human–computer interaction to one mode at a time and are designed with an assumption that use will be by individuals (rather than groups), directing (rather than interacting with) the system. To support the ways in which humans work and interact, a new paradigm for computing is required that is multimodal, rather than unimodal, collaborative, rather than personal, and dialogue‐enabled, rather than unidirectional. To address this challenge, we have developed an approach for designing natural, multimodal, multiuser dialogue‐enabled interfaces to geographic information systems that make use of large‐screen displays and integrated speech–gesture interaction. After outlining our goals and providing a brief overview of relevant literature, we introduce the Dialogue‐Assisted Visual Environment for Geoinformation (DAVE_G). DAVE_G is being developed using a human‐centred systems approach that contextualizes development and assessment in the current practice of potential users. In keeping with this human‐centred approach, we outline a user task analysis and associated scenario development that implementation is designed to support (grounded in the context of emergency response), review our own precursors to the current prototype system and discuss how the current prototype extends upon past work, provide a detailed description of the architecture that underlies the current system, and introduce the approach implemented for enabling mixed‐initiative human–computer dialogue. We conclude with a discussion of goals for future research.

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.

A novel interface system for accessing geospatial data (GeoMIP) has been developed that realizes a user-centered multimodal speech/gesture interface for addressing some of the critical needs in crisis management. In this system we primarily developed vision sensing algorithms, speech integration, multimodality fusion, and rule-based mapping of multimodal user input to GIS database queries. A demo system of this interface has been developed for the Port Authority NJ/NY and is explained here.

We demonstrate a same-time different-place collaboration system for managing crisis situations using geospatial information. Our system enables distributed spatial decision-making by providing a multimodal interface to team members. Decision makers in front of large screen displays and/or desktop computers, and emergency responders in the field with tablet PCs can engage in collaborative activities for situation assessment and emergency response.

Automatic initialization and tracking of multiple people and their body parts is one of the first steps in designing interactive multimedia applications. The key problems in this context are robust detection and tracking of people and their body parts in an unconstrained environment. This paper presents an integrated framework to address detection and tracking of multiple objects in a computationally efficient manner. In particular, a neural network-based face detector was employed to detect faces and compute person specific statistical model for skin color from the face regions. A probabilistic model was proposed to fuse the color and motion information to localize the moving body parts (hands). Multiple hypothesis tracking (MHT) algorithm was adopted to track face and hands. In real world scenes extracted features (face and hands) usually contain spurious measurements that create unconvincing trajectories and needless computations. To deal with this problem a path coherence function was incorporated along with MHT to reduce the number of hypotheses, which in turn reduces the computational cost and improves the structure of trajectories. The performance of the framework was validated using experiments on synthetic and real sequence of images.

Despite recent advances in gesture recognition, reliance on the visual signal alone to classify unrestricted continuous gesticulation is inherently error-prone. Since spontaneous gesticulation is mostly coverbal in nature, there have been some attempts of using speech cues to improve gesture recognition. Some attempts have been made in using speech cues to improve gesture recognition, e.g., keyword-gesture co-analysis. Use of such scheme is burdened by the complexity of natural language understanding. This paper offers a "signal-level" perspective by exploring prosodic phenomena of spontaneous gesture and speech co-production. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (co-articulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions in multimodal production, are exploited in an audio-visual feature integration framework using hidden Markov models (HMMs). Co-articulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.

This paper presents a systematic approach to acquiring model appearance information online for monocular model-based tracking. The acquired information is used to drive a set of complementary imaging cues to obtain a highly discriminatory observation model. Appearance is modeled as a Markov random field of color distributions over the model surface. The online acquisition process estimates appearance-based on uncertain image measurements and is designed to greatly reduce the chance of mapping non-object image data onto the model. Confidences about the different appearance driven imaging cues are estimated in order to adaptively balance the contributions of the different cues. The discriminatory power of the resulting model is good enough to allow long-duration single-hypothesis model-based tracking with no prior appearance information. Careful evaluation based on real and semi-synthetic video sequences shows that the presented algorithm is able to robustly track a wide variety of targets under challenging conditions.

Over the past years, video communication has emerged as one of the most popular form of communication. Developing the technology for merging the real and virtual image streams and 3D models for better communication is an active research topic. In this paper, we propose a novel method of real time augmentation of 3D objects in 3D scenes using trifocal tensors. The augmentation is robust to arbitrary camera motion. The proposed technique does not require camera calibration, use of fiducials and prior knowledge of structural model of the scene. Also, using background-foreground segmentation, the occlusion issue in augmented reality is addressed. We show several results of the successful working of our algorithm in real life situations. The technique works on a real time video from a USB camera, creative Webcam III on a P IV 1.6 GHz system without any special hardware support.

Computer vision systems for monitoring people and collecting valuable demographic information in a social environment is an important research problem. It is expected that such a system will play an increasingly important role in enhancing user's experience and can significantly improve the intelligibility of a human computer interaction (HCI) system. For example, a robust gender classification system can provide a basis for passive surveillance and access to a smart building using demographic information or can provide valuable consumer statistics in a public place. The option of an audio cue in addition to the visual cue promises a robust solution with high accuracy and ease-of-use in human computer interaction systems. This paper investigates gender classification using Support Vector Machines (SVMs). The visual (thumbnail frontal face) and the audio (features from speech data) cues were considered for designing the classifier. Three different representations of the data, namely, raw data, principle component analysis (PCA) and non-negative matrix factorization (NMF) were used for the experimentation with visual signal. For speech, mel-cepstral coefficient and pitch were used for the experimentation. It was found that the best overall classification rates obtained using the SVM for the visual and speech data were 95.31% and 100%, respectively, on data set collected in laboratory environment. The performance of the SVM was compared with two simple classifiers namely, the nearest prototype neighbor and the k-nearest neighbor on all feature sets. It was found that the SVM outperformed the other two classifiers on all datasets. To further understand the robustness issues, the proposed approach has been applied on a large balanced (roughly equal distribution of gender, ethnicity and age group) data-base consisting of 8000 faces collected in real world environment. While, the results are very promising it indicates more to be done to make a statistically meaningful conclusion.

Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE/spl I.bar/G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

Although recognition of natural speech and gestures have been studied extensively, previous attempts at combining them in a unified framework to boost classification were mostly semantically motivated, e.g., keyword-gesture co-occurrence. Such formulations inherit the complexity of natural language processing. This paper presents a Bayesian formulation that uses a phenomenon of gesture and speech articulation for improving accuracy of automatic recognition of continuous coverbal gestures. The prosodic features from the speech signal were co-analyzed with the visual signal to learn the prior probability of co-occurrence of the prominent spoken segments with the particular kinematical phases of gestures. It was found that the above co-analysis helps in detecting and disambiguating small hand movements, which subsequently improves the rate of continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected front the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.

This paper presents a framework for designing a natural multimodal human computer interaction (HCI) system. The core of the proposed framework is a principled method for combining information derived from audio and visual cues. To achieve natural interaction, both audio and visual modalities are fused along with feedback through a large screen display. Careful design along with due considerations of possible aspects of a systems interaction cycle and integration has resulted in a successful system. The performance of the proposed framework has been validated through the development of several prototype systems as well as commercial applications for the retail and entertainment industry. To assess the impact of these multimodal systems (MMS), informal studies have been conducted. It was found that the system performed according to its specifications in 95% of the cases and that users showed ad-hoc proficiency, indicating natural acceptance of such systems.

The ability to track many people and their body parts (i.e., face and hands) in a complex environment is crucial for designing collaborative natural human computer interaction (HCI). A challenging issue in tracking body parts is the data association uncertainty while assigning measurements to the proper tracks in the case of occlusion and close interaction of body parts of different people. This paper describes a framework for tracking body parts of people in 2D/3D using a multiple hypothesis tracking (MHT) algorithm. A path coherence function has been incorporated along with MHT to reduce the negative effects of closely spaced measurements that produce unconvincing tracks and unnecessary computations. The performance of the framework has been validated using experiments on a real sequence of images.

This paper presents a multimodal crisis management system (XISM). It employs processing of natural gesture and speech commands elicited by a user to efficiently manage complex dynamic emergency scenarios on a large display. The developed prototype system demonstrates the means of incorporating unconstrained free-hand gestures and speech in a real-time interactive interface. This paper provides insights into the design aspects of the XISM system. In particular, it addresses the issues of extraction and fusion of gesture and speech modalities to allow more natural interactive behavior. Performance characteristics of the current prototype and considerations for future work are discussed. A series of studies indicated positive response with respect to ease of interacting with the current system.

Tracking of people and their body parts has found application in image processing and computer vision. This paper describes a framework for tracking body parts (eg, hands and face) of multiple people in image sequences. We use a probabilistic model to fuse color and motion information to localize the body parts and employ the multiple hypothesis tracking (MHT) algorithm to track these features simultaneously. We incorporate a path coherence function along with MHT to reduce the negative effects of spurious measurements that produce unconvincing tracks and needless computations. The performance of the framework is validated using experiments on synthetic and real sequences of images.

In order to incorporate naturalness in the design of human computer interfaces (HCI), it is desirable to develop recognition techniques capable of handling continuous natural gesture and speech inputs. Though many different researchers have reported high recognition rates for gesture recognition using hidden Markov models (HMM), the gestures used are mostly pre-defined and are bound with syntactical and grammatical constraints. But natural gestures do not string together in syntactical bindings. Moreover, strict classification of natural gestures is not feasible. We have examined hand gestures made in a very natural domain, that of a weather person narrating in front of a weather map. The gestures made by the weather person are embedded in a narration. This provides us with abundant data from an uncontrolled environment to study the interaction between speech and gesture in the context of a display. We hypothesize that this domain is very similar to that of a natural human-computer interface. We present an HMM architecture for continuous gesture recognition framework and keyword spotting. To explore the relation between gesture and speech, we conducted a statistical co-occurrence analysis of different gestures with a selected set of spoken keywords. We then demonstrate how this co-occurrence analysis can be exploited to improve the performance of continuous gesture recognition.

Integration of visual sensing and motion planning can play a critical role in autonomous robot operation. We present a framework for vision-based robot motion planning that uses learning to handle arbitrarily configured cameras and robots. The theoretical basis of this approach is the concept of the perceptual control manifold (PCM) that extends the notion of the robot configuration space to include sensor space. This allows the inclusion of visual constraints in the motion planning. However, the analytical derivation of PCM is difficult in most cases and also depends on calibration of the camera. To overcome this modeling uncertainly, we propose the use of a topology representing network (TRN) to learn a suitable representation of the PCM. By exploiting the topology preserving features of the neural network, path planning strategies defined on the TRN lead to flexible obstacle avoidance. The practical feasibility of the approach is demonstrated by the results of simulation with a PUMA robot and experiments with a Mitsubishi robot.

This paper describes an interactive tool for evaluating assembly sequences using the novel human-computer interface of augmented reality. The goal is to be able to consider various sequencing alternatives of the manufacturing design process by manipulating both virtual and real prototype components. The augmented reality-based assembly evaluation tool would allow a manufacturing engineer to interact with the assembly planner while manipulating the real and virtual prototype components in an assembly environment. Information from the assembly planner can be displayed directly superimposed on the real. A sensing technique is proposed that uses computer vision along with a system of markers for automatically monitoring the assembly state as the user manipulates the assembly components. An implemented system called AREAS is described. The advantage of using mixed prototyping and augmented reality as means of capturing human intuition in assembly planning is also discussed.

The use of hard gestures provides an attractive means of interacting naturally with a computer generated display. Using one or more video cameras, the hand movements can potentially be interpreted as meaningful gestures. One key problem in building such am interface without a restricted setup is the ability to localize and track the human arm robust in image sequences. This paper proposes a multiple cite-based localization scheme combined with a tracking framework to reliably track the human arm dynamics in unconstrained environments. The localization scheme integrates the multiple cues of motion, shape, and color for locating a set of key image features. Using constraint fusion, these features are tracked by a modified Extended Kalman Filter that exploits the articulated structure of the arm. We also propose an interaction scheme between tracking and localization for improving the estimation process while reducing the computational requirements. The performance of the frameworks is validated with the help of extensive experiments and simulations.

Most current robotic systems treat motion planning and control as two independent subsystems where the motion planner determines a collision-free path for achieving a goal, and the control system attempts to follow the planned path using sensing. This may yield undesirable motion plans that do not utilize the sensing effectively or consider sensor constraints. In this paper we consider an approach for motion planning that incorporates visual servoing constraints into the computation of the motion plans. The approach extends the notion of configuration space to include the corresponding sensor values. We address the issue of representing the extended planning space and practical computational schemes for motion planning with visual servoing constraints. We propose a hierarchical representation of the high dimensional planning space involved, and a multi-strategic heuristics search. This results in a practical motion planning scheme that is proven to be resolution-complete.

Recent advances in various signal processing technologies, coupled with an explosion in the available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities: speech, vision-based gesture recognition, eye tracking, electroencephalograph, etc. Successful embodiment of these modalities into an interface has the potential of easing the HCI bottleneck that has become noticeable with the advances in computing and communication. It has also become increasingly evident that the difficulties encountered in the analysis and interpretation of individual sensing modalities may be overcome by integrating them into a multimodal human-computer interface. We examine several promising directions toward achieving multimodal HCI. We consider some of the emerging novel input modalities for HCI and the fundamental issues in integrating them at various levels, from early signal level to intermediate feature level to late decision level. We discuss the different computational approaches that may be applied at the different levels of modality integration. We also briefly review several demonstrated multimodal HCI systems and applications. Despite all the recent developments, it is clear that further research is needed for interpreting and fitting multiple sensing modalities in the context of HCI. This research can benefit from many disparate fields of study that increase our understanding of the different human communication modalities and their potential role in HCI.

Knowledge of the complex molecular structures of living cells is being accumulated at a tremendous rate. Key technologies enabling this success have been, high performance computing and powerful molecular graphics applications, but the technology is beginning to seriously lag behind challenges posed by the size and number of new structures and by the emerging opportunities in drug design and genetic engineering. A visual computing environment is being developed which permits interactive modeling of biopolymers by linking a 3D molecular graphics program with an efficient molecular dynamics simulation program executed on remote high-performance parallel computers. The system will be ideally suited for distributed computing environments, by utilizing both local 3D graphics facilities and the peak capacity of high-performance computers for the purpose of interactive biomolecular modeling. To create an interactive 3D environment three input methods will be explored: (1) a six degree of freedom "mouse" for controlling the space shared by the model and the user; (2) voice commands monitored through a microphone and recognized by a speech recognition interface; (3) hand gestures, detected through cameras and interpreted using computer vision techniques. Controlling 3D graphics connected to real time simulations and the use of voice with suitable language semantics, as well as hand gestures, promise great benefits for many types of problem solving environments. Our focus on structural biology takes advantage of existing sophisticated software, provides concrete objectives, defines a well-posed domain of tasks and offers a well-developed vocabulary for spoken communication.

The calibration requirements for visual servoing can make it difficult to apply in many real-world situations. One approach to image-based visual servoing without calibration is to dynamically estimate the image Jacobian and use it as the basis for control. However, with the normal motion of a robot toward the goal, the estimation of the image Jacobian deteriorates over time. The authors propose the use of additional exploratory motion to considerably improve the estimation of the image Jacobian. They study the role of such exploratory motion in a visual servoing task. Simulations and experiments with a 6-DOF robot are used to verify the practical feasibility of the approach.

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient "purposive" approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction.

We propose a novel neural network, called the self-organized invertible map (SOIM), that is capable of learning many-to-one functionals mappings in a self-organized and online fashion. The design and performance of the SOIM are highlighted by learning a many-to-one functional mapping that exists in active vision for spatial representation of three-dimensional point targets. The learned spatial representation is invariant to changing camera configurations. The SOIM also possesses an invertible property that can be exploited for active vision. An efficient and experimentally feasible method was devised for learning this representation on a real active vision system. The proof of convergence during learning as well as conditions for invariance of the learned spatial representation are derived and then experimentally verified using the active vision system. We also demonstrate various active vision applications that benefit from the properties of the mapping learned by SOIM.

We address the ability of a computer vision system to perceive the motion of an object (possibly a robot manipulator) in its field of view. We derive a quantitative measure of motion perceptibility, which relates the magnitude of the rate of change in an object's position to the magnitude of the rate of change in the image of that object. We then show how motion perceptibility can be combined with the traditional notion of manipulability, into a composite perceptibility/manipulability measure. We demonstrate how this composite measure may be applied to a number of different problems involving relative hand/eye positioning and control.

Visual feedback can play a crucial role in a dynamic robotic task such as the interception of a moving target. To utilize the feedback effectively, there is a need to develop robot motion planning techniques that also take into account properties of the sensed data. We propose a motion planning framework that achieves this with the help of a space called the perceptual control manifold (PCM) defined on the product of the robot configuration space and an image-based feature space. We show how the task of intercepting a moving target can be mapped to the PCM, using image feature trajectories of the robot end-effector and the moving target. This leads to the generation of motion plans that satisfy various constraints and optimality criteria derived from the robot kinematics, the control system, and the sensing mechanism. Specific interception tasks are analyzed to illustrate this vision-based planning technique.

In recent years there has been tremendous progress in 3-D, immersive display and virtual reality (VR) technologies. Scientific visualization af data is one of many applications that has benefited from this progress. To fully exploit the potential of these applications in the new environment there is a need for "natural" interfaces that allow the manipulation of such displays without burdensome attachments. This paper describes the use of visual hand gesture analysis enhanced with speech recognition for developing a bimodal gesture/speech interface for controlling a 3-D display. The interface augments an existing application, VMD, which is a VR visual computing environment for molecular biologists. The free hand gestures are used for manipulating the 3-D graphical display together with a set of speech commands. We concentrate on the visual gesture analysis techniques used in developing this interface. The dual modality of gesture/speech is found to greatly aid the interaction capability.

Recent progress in 3-D, immersive display and virtual reality (VR) technologies has made possible many exciting applications, for example interactive visualization of complex scientific data. To fully exploit this potential there is a need for "natural" interfaces that allow the manipulation of such displays without cumbersome attachments. In this paper we describe the use of visual hand gesture analysis and speech recognition for developing a speech/gesture interface for controlling a 3-D display. The interface enhances an existing application, VMD, which is a VR visual computing environment far molecular biologists. The free hand gestures are used for manipulating the 3-D graphical display together with a set of speech commands. We describe the visual gesture analysis and the speech analysis techniques used in developing this interface. The dual modality of speech/gesture is found to greatly aid the interaction capability.

Robots that use an active camera system for visual feedback can achieve greater flexibility, including the ability to operate in a dynamically changing environment. Incorporating active vision into a robot control loop involves some inherent difficulties, including calibration, and the need for redefining the goal as the camera configuration changes. In this paper, we propose a novel self-organizing neural network (SOIM) that learns a calibration-free spatial representation of 3D point targets in a manner that is invariant to changing camera configurations. This representation is used to develop a new framework for robot control with active vision. The salient feature of this framework is that it decouples active camera control from robot control. The feasibility of this approach is explored with the help of computer simulations and experiments with the University of Illinois Active Vision System (UIAVS).

Visual feedback can play a crucial role in a dynamic robotic task such as the interception of a moving target. To utilize the feedback effectively, there is a need to develop robot motion planning techniques that also take into account properties of the sensed data. We propose a motion planning framework that achieves this with the help of a space called the perceptual control manifold (PCM) defined on the product of the robot configuration space and an image-based feature space. We show how the task of intercepting a moving target can be mopped to the PCM. This leads to the generation of motion plans that satisfy various constraints and optimality criteria derived from the robot kinematics, the control system, and the sensing mechanism.

Gross-motion planning for assembly is commonly considered as a distinct, isolated step between task sequencing/scheduling and fine-motion planning. In this paper the authors formulate a problem of delivering parts for assembly in a manner that integrates it with both the manufacturing process and the fine motions involved in the final assembly stages. One distinct characteristic of gross-motion planning for assembly is the prevalence of uncertainty involving time-in parts arrival, in request arrival, etc. The authors propose a stochastic representation of the assembly process, and design a state-feedback controller that optimizes the expected time that parts wait to be delivered. This leads to increased performance and a greater likelihood of stability in a manufacturing process. Six specific instances of the general framework are modeled and solved to yield optimal motion strategies for different robots operating under different assembly situations. Several extensions are also discussed.

An important initial step in interpreting a dynamic scene is to detect moving objects in the environment. This paper presents a novel solution to the problem of early motion detection by a moving observer. The solution requires the observer to be active in the acquisition of images thereby controlling the optical flow pattern due to egomotion. A theoretical analysis is done based on geometric considerations to establish conditions that are necessary and sufficient to guarantee motion detection at a point. The detection problem is posed in terms of locally computable image quantities (the normal image flow) which this makes it implementable in real time. The performance of the technique can be improved by imposing any applicable constraint; this is demonstrated for the detection of the motions of "compact" objects satisfying a size bound. The goal is to design a flexible and efficient early motion detection strategy that can be tailored to the needs of a particular navigation system.

The authors (1994) derived a quantitative measure of the ability of a camera setup to observe the changes in image features due to relative motion. This measure of motion perceptibility has many applications in evaluating a robot hand/eye setup with respect to the ease of achieving vision-based control, and steering away from singular-configurations. Motion perceptibility can be combined with the traditional notion of manipulability, into a composite perceptibility/manipulability measure. In this paper the authors demonstrate how this composite measure may be applied to a number of different problems involving relative hand/eye positioning and control. These problems include optimal camera placement, active camera trajectory planning, robot trajectory planning, and feature selection for visual servo control. The authors consider the general formulation of each of these problems, and several others, in terms of the motion perceptibility/manipulability measure and illustrate the solution for particular hand/eye configurations.

Presents a framework for analyzing and determining motion plans for a robot that operates in an environment that changes over time in an uncertain manner. The authors first classify sources of uncertainty in motion planning into four categories, and argue that the framework addressed in this paper characterizes an important, yet little-explored category. The authors treat the changing environment in a flexible manner by combining traditional configuration space concepts with a Markov process that models the environment. For this context, the authors then propose the use of a motion strategy, which provides a motion command for the robot for each contingency that it could be confronted with. The authors allow the specification of a desired performance criterion, such as time or distance, and the goal is to determine a motion strategy that is optimal with respect to that criterion. A motion planning problem in this framework is formulated as the design of a stochastic optimal controller.

The authors (1995) have previously presented a framework for analyzing motion plans for a robot that operates in an environment that changes over time in an uncertain manner. In this paper, the authors demonstrate the utility of their framework by applying it to a variety of motion planning problems. Examples are computed for problems that involve a changing configuration space, hazardous regions and shelters, and processing of random service requests. To achieve this, the authors have exploited the powerful principle of optimality, which leads to a dynamic programming-based algorithm for determining optimal strategies. Several computed examples are presented and discussed.

Defines a measure of "observability" of robot motion that can be used in evaluating a hand/eye set-up with respect to the ease of achieving vision-based control. This extends the analysis of "manipulability" of a robotic mechanism in Yoshikawa (1983) to incorporate the effect of visual features. The authors discuss how the observability measure can be applied for active camera placement and for robot trajectory planning to improve the visual servo control. The authors use the examples of a planar 2-DOF arm and a PUMA-type 3-DOF arm to show the variation of the observability and manipulability measure with respect to the relative position of the active camera.>

This paper describes two case studies of motion planning that uses a space called Perceptual Control Surface (PCS) introduced previously as a basis for visually controlling a robot manipulator. The motion planning problems discussed here are limited to dynamic robot hand positioning for interception tasks. Using this approach, a motion planning problem can be regarded as finding a path in robot's PCS which satisfies certain criteria. Qualitatively different tasks correspond to different type of paths in the PCS. These correspondences are discussed and techniques to find these different paths are outlined.>

The problem of efficient path planning for a point robot in a partially known dynamic environment is considered. The static known part of the environment consists of point shelters distributed in planar terrain, and the dynamic, unknown part is abstracted in the form of alarms that cause the robot to leave its current (preplanned) path and divert to the nearest shelter. We give a probabilistic analysis of the expected times for the dynamic paths generated when the alarms follow a Poisson distribution with parameter lambda . A case study with three shelters serves to illustrate the dependence of the expected travel times on lambda for two alternate static paths. Two different strategies are presented for the general case of n shelters and shown to be superior for different ranges of values of the alarm rate lambda (very low and very high values, respectively). We also discuss some ways of generalizing the approach and possible applications.>

The development of the visual processes that would be needed by a mobile robot system for visually intercepting a moving target is considered. Many relevant active visual processes are proposed that provide robust input for qualitative motion control strategies. The processes for detecting independent motion and for monitoring progress toward the moving target are summarized.>

Current approaches to robot motion planning are limited in their ability to deal with an uncertain and dynamically changing environment. The author discusses the difficulties involved in modeling the situation and proposes a probabilistic model based on discrete events that abstracts the dynamic interaction between the mobile robot and the unknown part of the environment. The resulting framework makes it possible to design and evaluate motion planning strategies that consider both the known portion of the environment and the portion that is unknown but satisfies a probability distribution. Studies of instances of the general model that have yielded useful results in designing efficient motion planning algorithms vis-a-vis parameters representing a robot's environment and the reactive component of its behavior are summarized.>

The authors consider the multiple applications of vision in the solution of dynamic robot and positioning problems such as catching, hitting, and intercepting a moving object. They propose a framework for qualitative reasoning about such problems, and then present a robust vision-based control strategy that uses a Kalman filter to track image features. In the formulation, a given positioning problem is transformed into one of constrained trajectory planning on a topological surface defined by the relationship between positions of the robot and observable image features. By considering only the qualitative properties of this surface, the need for precise calibration of the various components of the system is eliminated.>

This paper presents an active vision technique for monitoring the time-to-collision to a moving or stationary point in the scene from a mobile robot platform carrying a monocular vi- sion system that tracks the point. This can pro- vide a loosely coupled control feedback for an intelligent motion control strategy used to move toward that target point. The vision technique presented is robust and completely paralleliz- able, being based on just the spatio- temporal gradients of the image.

The problem addressed is that of efficiently planning a path for a robot between two points when the path is forced to change dynamically by the occurrence of certain events in the environment. An event or an alarm, for example, may be the discovery of another moving object on a collision course with the robot and would require some evasive action. A probabilistic model is given that represents the robot's dynamic behavior in response to alarms that have a Poisson distribution, and safety rules that assume that some regions are safe. A provably optimal expected solution for the problem is given, and the variation of the optimal path with two parameters that represent the alarm rate and the safety rule, respectively, is discussed.>

The warehouseman's problem, namely, the coordinated motion planning for multiple independent objects confined in a room, is addressed. Several constraints are presented under which the problem becomes tractable, and polynomial time algorithms that guarantee a coordinated rearrangement under the given conditions are obtained. The concept of temporary storage space (TSS) is introduced as a general way of constraining free space. Each algorithm presented has a different set of constraints on the possible sizes and/or relative placements of the square blocks. For each case, an adequate TSS is proposed that guarantees rearrangement of n blocks through algorithms having O(n/sup 2/) running time. The practical utility of the presented techniques is also discussed in the light of the complexity of motion coordination.>

The problem of efficiently planning a path for a robot between two points when the path is forced to change dynamically by the occurrence of certain events in the environment is addressed. An event, for example, may be the discovery of another moving object on a collision course with the robot. The robot is forced to take evasive action whenever such an alarm occurs. A probabilistic model that represents the robot's dynamic behavior in response to alarms with a Poisson distribution and safety rules that assume that some regions are safe are presented. A provably optimal expected solution for the problem is presented, and the role of the probabilistic parameter of the dynamic environment on the optimal path, and the role of vision (or time to collision) on the planned paths are studied. The simplicity of the model used is justified by the difficulty of analyzing a more complicated (unknown) dynamic environment, and by the generality of the results obtained using this simple model.>