The goal of this work is to detect pairwise primitive interactions in groups for social interaction analysis in real environments. We propose a system that extracts locations and head poses of people from videos captured in an unconstrained environment, namely a research lab. Our system is designed to work with realistic data capturing natural human interactions. An efficient tracking method based on Chamfer matching finds the head and shoulder silhouettes of people in real-time, and a head orientation classifier estimates their head poses. The location, relative distance and head orientation of people capture the use of space by individuals and their interactive behavioral patterns which are inferred with a probabilistic model. We present quantitative evaluation and experimental results of our system, demonstrating the effectiveness of our proposed approach on challenging real-world data.

This demonstration presents a social interaction analysis system designed to operate in real-time and under real environment conditions. A webcam is used to capture videos of a group of people interacting in an unconstrained environment. Locations of multiple people and their head poses are extracted from the videos. Direct pairwise interactions are detected based on the relative distance and head orientation using a probabilistic model. Communicative behavior analysis is carried out by grouping people based on their interaction patterns, i.e. frequency and duration, among the social group.

We propose a data integration and reasoning technique for automatic environment discovery in smart homes based on observations of user interactions with objects. This approach is complementary to traditional appearance-based object recognition, which often demands large training sets. In our approach, object recognition is achieved in a semantic way by linking object behaviors to the pose and activity of the person using them. The complex relations between objects and user activities are modeled with a Markov logic network. The embodiments of the proposed approach in two multicamera smart environments are described, and the experimental results are presented.

While providing a variety of intriguing application opportunities, a vision sensor network poses three key challenges. High computation capacity is required for early vision functions to enable real-time performance. Wireless links limit image transmission in the network due to both bandwidth and energy concerns. Last but not least, there is a lack of established vision-based fusion mechanisms when a network of cameras is available. In this paper a distributed vision processing implementation of human pose interpretation on a wireless smart camera network is presented. The motivation for employing distributed processing is to both achieve real-time vision and provide scalability for developing more complex vision algorithms. The distributed processing operation includes two levels. One is that each smart camera processes its local vision data, achieving spatial parallelism. The other is that different functionalities of the whole line of vision processing are assigned to early vision and object-level processors, achieving functional parallelism based on the processor capabilities. Aiming for low power consumption and high image processing performance, the wireless smart camera is based on an SIMD (single-instruction multiple-data) video analysis processor, an 8051 micro-controller as the local host, and wireless communication through the IEEE 802.15.4 standard. The vision algorithm implements 3D human pose reconstruction. From the live image data from the sensor the smart camera acquires critical joints of the subject in the scene through local processing. The results obtained by multiple smart cameras are then transmitted through the wireless channel to a central PC where the 3D pose is recovered and demonstrated in a virtual reality gaming application. The system operates in real time with a 30 frames/sec rate.

Monitoring human activities finds novel applications in smart environment settings. Examples include immersive multimedia and virtual reality, smart buildings and occupancy-based services, assisted living and patient monitoring, and interactive classrooms and teleconferencing. A network of cameras can enable detection and interpretation of human events by utilizing multiple views and collaborative processing. Distributed processing of acquired videos at the source camera facilitates operation of scalable vision networks by avoiding transfer of raw images. This allows for efficient collaboration between the cameras under the communication and latency constraints, as well as being motivated by aiming to preserve the privacy of the network users (no image transfer out of the camera) while offering services in applications such as assisted living or virtual placement. In this paper, collaborative processing and data fusion techniques in a multicamera setting are examined in the context of human pose estimation. Multiple mechanisms for information fusion across the space (multiple views), time, and different feature levels are introduced to meet system constraints and are described through examples.

In multi-camera networks rich visual data is provided both spatially and temporally. In this paper a method of human posture estimation is described incorporating the concept of an opportunistic fusion framework aiming to employ manifold sources of visual information across space, time, and feature levels. One motivation for the proposed method is to reduce raw visual data in a single camera to elliptical parameterized segments for efficient communication between cameras. A 3D human body model is employed as the convergence point of spatiotemporal and feature fusion. It maintains both geometric parameters of the human posture and the adaptively learned appearance attributes, all of which are updated from the three dimensions of space, time and features of the opportunistic fusion. In sufficient confidence levels parameters of the 3D human body model are again used as feedback to aid subsequent in-node vision analysis. Color distribution registered in the model is used to initialize segmentation. Perceptually Organized Expectation Maximization (POEM) is then applied to refine color segments with observations from a single camera. Geometric configuration of the 3D skeleton is estimated by Particle Swarm Optimization (PSO).

A layered and collaborative architecture for gesture recognition in a multi-camera network is presented in this paper. The proposed approach is motivated by the diversity of gestures expressed in passive monitoring applications. It is based on the concept of opportunistic fusion of simple features within a single camera and active collaboration between multiple cameras in the decision making process. The decision process is pursued through mutual, assisted, and self correspondences, using features available at different cameras. The dynamics employed by the opportunistic fusion of different features within a single camera as well as those from multiple cameras offer the potential to address gesture recognition problems more efficiently and accurately across a variety of different applications.