Maintaining formation integrity is a significant challenge for a multi-agent system while navigating in cluttered environments. This paper presents a multi-agent formation control, integrating a virtual leader strategy with formation switching (FS) and Safe Artificial Potential Field (SAPF) control. Our goal is to navigate an Unmanned Aerial Vehicle (UAV) formation collision-free around obstacles allowing smooth transitions among three formations based on the available space. Simulation results show that our FS-SAPF framework achieves a higher success rate by effectively handling local minima and reducing oscillations than traditional Artificial Potential Field (APF) approaches. Our FS-SAPF maintains a larger minimum distance between agents and obstacles, thus enhancing safety in complex environments.

Multi-human parsing algorithms have significant potential for real-time surveillance applications. By accurately segmenting humans and their body parts, such algorithms help to understand and better differentiate multiple human subjects in video frames. However, deploying such algorithms on resource-constrained embedded devices such as smart cameras presents challenges due to memory constraints and limited computational power. Therefore, this work investigates the limitations of existing multi-human parsing algorithms and proposes MHParsNet, a lightweight yet accurate model for human parsing on embedded devices. Compared to benchmark algorithms, MHParsNet performs competitive segmentation while requiring only 125 MB of memory. We deployed MHParsNet on a smart camera prototype using the Jetson Nano embedded board and achieved an average inference of 6 frames per second. These results demonstrate the effectiveness of MHParsNet and its suitability for real-time applications on resource-constrained embedded devices.

Small unmanned aerial vehicles, commonly called drones, enable novel applications in many domains. Multidrone systems are a current key trend where several drones operate collectively as an integrated networked autonomous system to complete various missions. The specification and execution of multidrone missions are particularly challenging, since substantial expertise of the mission domain, the drone's capabilities, and the drones' software environment is required to properly encode the mission. in this position paper, we introduce a specification language for multidrone missions and describe the transcoding of its components into the multidrone execution environment for both simulations and real drones. The key features of our approach include (i) domain-independence of the mission specification, (ii) readability and ease of use, and (iii) expandability. The specification language has a simple syntax and uses a parameterized description of execution blocks and mission capabilities, which are derived from native drone functions. Domain-independence and expandability are provided by a clear separation between the specification and the implementation of the mission tasks. We demonstrate the effectiveness of our approach with a selected multidrone mission example.

Planning the simultaneous movement of multiple agents represents a challenging coordination problem, and ideally safety and efficiency are jointly addressed. This paper introduces a planning algorithm for fast and energy-efficient trajectories with reduced collision potential from a start to an end constellation. This new approach combines trajectory approximation based on model predictive control, collision avoidance with potential fields, and flight energy optimization with minimum snap trajectories. Our approach results in unprecedented transition times and success rates with less energy consumption, as shown in simulation and real experiments with 16 drones.

The articles in this month’s special issue cover concepts and fundamentals, architectures and techniques, and applications and case studies in the exciting area of self-awareness in autonomous systems.

In a computational context, self-awareness (SA) is a capability of an autonomous system to describe the acquired experience about itself and its surrounding environment with appropriate models and correlate them incrementally with the currently perceived situation to expand its knowledge continuously. This article introduces a bio-inspired framework for generative and descriptive dynamic models that support SA computationally and efficiently. Generative models facilitate predicting future states, while descriptive models enable the selection of the representation that best fits the current observation. Our framework is founded on the analysis and extension of three bio-inspired theories that have studied SA from different viewpoints, and we demonstrate how probabilistic techniques, such as cognitive dynamic Bayesian networks and generalized filtering paradigms, can learn appropriate models from multidimensional proprioceptive and exteroceptive signals acquired by the autonomous system. We discuss essential capabilities for SA and show how our modeling framework supports these capabilities in theory and through a case study where a mobile robot uses multisensorial data to determine its internal and environmental state as well as distinguishing among normal and abnormal behaviors.

Today, camera networks are pervasively used in smart environments such as intelligent homes, industrial automation or surveillance. These applications often require cameras to be aware of their spatial neighbors or even to operate on a common ground plane. A major concern in the use of sensor networks in general is their robustness and reliability even in the presence of attackers. This paper addresses the challenge of detecting malicious nodes during the calibration phase of camera networks. Such a resilient calibration enables robust and reliable localization results and the elimination of attackers right after the network deployment. Specifically, we consider the problem of identifying subverted nodes which manipulate calibration data and can not be detected by standard cryptographic methods. The experiments in our network show that our self-calibration algorithm enables location-unknown cameras to successfully detect malicious nodes while autonomously calibrating the network.

Future Internet of Things (IoT) applications will face challenges in increased flexibility, uncertainty, dynamics and scalability. Self-aware computing maintains knowledge about the applications state and environment and then uses this knowledge to reason about and adapt behaviours. In this position paper, we introduce self-aware computing as design approach for IoT applications which is centred around a self-aware architecture for IoT nodes. This architecture particularly supports adaptations based on node interactions. We demonstrate our approach with an IoT case study on multi-object coverage with mobile cameras.

In a goal-oriented Human Robot Collaborative (HRC) scenario, where the goal is to complete an assembly process, a robust object tracker might not necessarily fulfill its functional role due to the dynamic nature of HRC. Moreover, for an efficient HRC, the functional role of the object tacker should not only be limited to localizing and tracking objects for robotic manipulation. It should also help to determine the current state of the assembly process and verify if the chosen action has been successfully performed and thus to enable an uninterrupted completion of an HRC assembly process. We present a Context Enhanced Framework for Multi Object Tracking, that i) allows uninterrupted completion of an assembly process, ii) improves the overall functional accuracy of the object tracker from 49 percent to 96 percent, and iii) enables the object tracker to handle multiple instance of multiple objects in a HRC setting.

Recreational videography from small drones can capture bystanders who may be uncomfortable about appearing in those videos. Existing privacy filters, such as scrambling and hopping blur, address this issue through de-identification but generate temporal distortions that manifest themselves as flicker. To address this problem, we present a robust spatiotemporal hopping blur filter that protects privacy through de-identification of face regions. The proposed filter is meant for on-board installation and produces temporally smooth and pleasant videos. We apply hopping blur to protect each frame against identification attacks, and minimise artefacts and flicker introduced by the hopping blur. We evaluate the proposed filter against different identification attacks and by assessing the quality of the resulting videos using a subjective test and objective measures.

Ubiquitous and networked sensors impose a huge challenge for privacy protection which has become an emerging problem of modern society. Protecting the privacy of visual data is particularly important due to the omnipresence of cameras, and various protection mechanisms for captured images and videos have been proposed. This paper introduces an objective evaluation framework in order to assess such protection methods. Visual privacy protection is typically realised by obfuscating sensitive image regions which often results in some loss of utility. Our evaluation framework assesses the achieved privacy protection and utility by comparing the performance of standard computer vision tasks, such as object recognition, detection and tracking on protected and unprotected visual data. The proposed framework extends the traditional frame-by-frame evaluation approach by introducing two new approaches based on aggregated and fused frames. We demonstrate our framework on eight differently protected video-sets and measure the trade-off between the improved privacy protection due to obfuscating captured image data and the degraded utility of the visual data. Results provided by our objective evaluation method are compared with an available state-of-the-art subjective study of these eight protection techniques.

The deployment of multiple robots for achieving a common goal helps to improve the performance, efficiency, and/or robustness in a variety of tasks. In particular, the observation of moving targets is an important multirobot application that still exhibits numerous open challenges, including the effective coordination of the robots. This paper reviews control techniques for cooperative mobile robots monitoring multiple targets. The simultaneous movement of robots and targets makes this problem particularly interesting, and our review systematically addresses this cooperative multirobot problem for the first time. We classify and critically discuss the control techniques: cooperative multirobot observation of multiple moving targets, cooperative search, acquisition, and track, cooperative tracking, and multirobot pursuit evasion. We also identify the five major elements that characterize this problem, namely, the coordination method, the environment, the target, the robot and its sensor(s). These elements are used to systematically analyze the control techniques. The majority of the studied work is based on simulation and laboratory studies, which may not accurately reflect real-world operational conditions. Importantly, while our systematic analysis is focused on multitarget observation, our proposed classification is useful also for related multirobot applications.

In this paper we present a distributed, autonomous network calibration algorithm, which enables visual sensor networks to gather knowledge about the network topology. A calibrated sensor network provides the basis for more robust applications, since nodes are aware of their spatial neighbors. In our approach, sensor nodes estimate relative positions and orientations of nodes with overlapping fields of view based on jointly detected objects and geometric relations. Distance and angle measurements are the only information required to be exchanged between nodes. The process works iteratively, first calibrating camera neighbors in a pairwise manner and then spreading the calibration information through the network. Further, each node operates within its local coordinate system avoiding the need for any global coordinates. While existing methods mostly exploit computer vision algorithms to relate nodes to each other based on their images, we solely rely on geometric constraints.

The strong resource limitations of unmanned aerial vehicles (UAVs) pose various challenges for UAV applications. In persistent multi-UAV surveillance, several UAVs with limited communication range and flight time have to repeatedly visit sensing locations while maintaining a multi-hop connection to the base station. In order to achieve persistence, the UAVs need to fly back to the base station in time for recharge. However, simple motion planning algorithms can result in mutual movement obstructions of UAVs caused by the constraints. We introduce two planning algorithms with different planning horizons and cooperation and compare their performance in simulation studies. It can be seen that the short horizon uncooperative strategy can outperform other strategies if a sufficient number of UAVs is used. The full horizon strategy can generate a solution visiting all sensing locations if the existence conditions for such a solution are fulfilled.

Cameras and other sensors are increasingly deployed for private space monitoring applications such as home monitoring, assisted/enhanced living and child monitoring. Since these cameras capture highly sensitive information and transfer it over public communication infrastructures, security and privacy is a major concern. This work presents a secure camera device along with a secure data delivery and archiving solution for private space monitoring applications using untrusted public cloud storage services. Integrity, authenticity, confidentiality and freshness of captured data are protected on-board using physically unclonable functions (PUF). The protection holds true for entire lifetime of the data until it is consumed by an authorized end-user. Experimental results obtained from our Zynq7010 SoC based prototype shows that the device is able to secure videos with 30 frames per second at 640×480 resolution with marginal overhead. The presented solution is not limited to visual sensing but can be applied to a wide range of pervasive sensing and secure data delivery scenarios.

Airborne cameras on low-flying unmanned vehicles introduce new privacy challenges due to their mobility and viewing angles. In this paper, we focus on face recognition from airborne cameras and explore the design space to determine when a face in an airborne image is inherently protected, that is when an individual is not recognizable. Moreover, when individuals are recognizable by facial recognition algorithms, we propose an adaptive filtering mechanism to lower the face resolution in order to preserve privacy while ensuring a minimum reduction of the fidelity of the image. In particular, we estimate the resolution of faces captured at different altitudes and tilt angles using the data from navigation sensors and ascertain when the captured face is inherently protected. When the face is unprotected, we define a mechanism that automatically configures the strength of a privacy protection filter to improve the trade-off between privacy protection and fidelity of an aerial image or video.

There is a clear trend in camera networks toward enhanced functionality and flexibility, and a fixed static deployment is typically not sufficient to fulfill these increased requirements. Dynamic network reconfiguration helps to optimize the network performance to the currently required specific tasks while considering the available resources. Although several reconfiguration methods have been recently proposed, e.g., for maximizing the global scene coverage or maximizing the image quality of specific targets, there is a lack of a general framework highlighting the key components shared by all these systems. In this paper, we propose a reference framework for network reconfiguration and present a short survey of some of the most relevant state-of-the-art works in this field, showing how they can be reformulated in our framework. Finally, we discuss the main open research challenges in camera network reconfiguration.

This paper presents a centralized algorithm for multi-scale observation of multiple moving targets using a team of Micro Aerial Vehicles (MAVs). The proposed algorithm is appropriate when MAVs can observe targets at different elevations with the objective of jointly maximizing duration and resolution of observation for each target. The MAVs share the workload using a greedy assignment of locations and targets to MAVs. The proposed algorithm uses a quad-tree data structure to model the movement decisions of MAVs as well as the variable qualities (resolutions) of observations. We consider cases where there is uncertainty in the target observations (i.e., measurement noise), the number of targets is larger than that of the MAVs and the combined field of views (FOVs) of the sensors cannot cover the whole search region. Simulation results confirm the effectiveness of the proposed algorithm.

We study heterogeneity among nodes in self-organizing smart camera networks, which use strategies based on social and economic knowledge to target communication activity efficiently. We compare homogeneous configurations, when cameras use the same strategy, with heterogeneous configurations, when cameras use different strategies. Our first contribution is to establish that static heterogeneity leads to new outcomes that are more efficient than those possible with homogeneity. Next, two forms of dynamic heterogeneity are investigated: nonadaptive mixed strategies and adaptive strategies, which learn online. Our second contribution is to show that mixed strategies offer Pareto efficiency consistently comparable with the most efficient static heterogeneous configurations. Since the particular configuration required for high Pareto efficiency in a scenario will not be known in advance, our third contribution is to show how decentralized online learning can lead to more efficient outcomes than the homogeneous case. In some cases, outcomes from online learning were more efficient than all other evaluated configuration types. Our fourth contribution is to show that online learning typically leads to outcomes more evenly spread over the objective space. Our results provide insight into the relationship between static, dynamic, and adaptive heterogeneity, suggesting that all have a key role in achieving efficient self-organization.

Images captured in camera networks are potentially privacy sensitive and therefore need protection. A critical aspect is where protection is applied - after transmission at the data center or preferably already on the camera. In this work we take on-camera protection a step further and propose to make privacy protection and security inherent features of the image sensing unit. Already within the camera we realize strong separation between components that have access to raw image data and those that do not need raw data access. Our approach is based on the custom-designed TrustEYE.M4 prototype of a secure sensing unit. We demonstrate the feasibility of sensor-level privacy protection with a cartoon-like effect based on mean shift filtering.

Visual privacy in video-based applications such as surveillance, assisted living or home monitoring is a highly active research topic. It is critical to protect the privacy of monitored people without severely limiting the utility of the system. We present a resource-aware cartooning privacy protection filter which converts raw images into abstracted frames where the privacy revealing details are re-moved. Cartooning can be applied either to entire images or pre-selected sensitive regions of interest. We provide an adaptation mechanism to our cartooning technique where the operator can easily change the filter intensity. The feasibility of this new approach is demonstrated by its deployment to real-world embedded smart cameras. We evaluate privacy protection and utility of cartooning with the PEViD data set and compare it with the two widely-used privacy filters: blurring and pixelation.

We present an end-to-end approach for trajectory clustering from aerial videos that enables the extraction of motion patterns in urban scenes. Camera motion is first compensated by mapping object trajectories on a reference plane. Then clustering is performed based on statistics from the Discrete Wavelet Transform coefficients extracted from the trajectories. Finally, motion patterns are identified by distance minimization from the centroids of the trajectory clusters. The experimental validation on four datasets shows the effectiveness of the proposed approach in extracting trajectory clusters. We also make available two new real-world aerial video datasets together with the estimated object trajectories and ground-truth cluster labeling.

In this paper, we propose strategies for merging occupancy probabilities of target existence in multi-UAV cooperative search. The objective is to determine the impact of cooperation and type of information exchange on search time and detection errors. To this end, we assume that small-scale UAVs (e.g., quadrotors) with communication range limitations move in a given search region following pre-defined paths to locate a single stationary target. Local occupancy grids are used to represent target existence, to update its belief with local observations and to merge information from other UAVs. Our merging strategies perform Bayes updates of the occupancy probabilities while considering realistic limitations in sensing, communication and UAV movement - all of which are important for small-scale UAVs. Our simulation results show that information merging achieves a reduction in mission time from 27% to 70% as the number of UAVs grows from 2 to 5.

Dynamic power management (DPM) refers to strategies which selectively change the operational states of a device during runtime to reduce the power consumption based on the past usage pattern, the current workload, and the given performance constraint. The power management problem becomes more challenging when the workload exhibits nonstationary behavior which may degrade the performance of any single or static DPM policy. This article presents a reinforcement learning (RL)-based DPM technique for optimal selection of timeout values in the different device states. Each timeout period determines how long the device will remain in a particular state before the transition decision is taken. The timeout selection is based on workload estimates derived from a Multilayer Artificial Neural Network (ML-ANN) and an objective function given by weighted performance and power parameters. Our DPM approach is further able to adapt the power-performance weights online to meet user-specified power and performance constraints, respectively. We have completely implemented our DPM algorithm on our embedded traffic surveillance platform and performed long-term experiments using real traffic data to demonstrate the effectiveness of the DPM. Our results show that the proposed learning algorithm not only adequately explores the power-performance trade-off with nonstationary workload but can also successfully perform online adjustment of the trade-off parameter in order to meet the user-specified constraint.

In this article we present an approach to object tracking handover in a network of smart cameras, based on self-interested autonomous agents, which exchange responsibility for tracking objects in a market mechanism, in order to maximise their own utility. A novel ant-colony inspired mechanism is used to learn the vision graph, that is, the camera neighbourhood relations, during runtime, which may then be used to optimise communication between cameras. The key benefits of our completely decentralised approach are on the one hand generating the vision graph online, enabling efficient deployment in unknown scenarios and camera network topologies, and on the other hand relying only on local information, increasing the robustness of the system. Since our market-based approach does not rely on a priori topology information, the need for any multicamera calibration can be avoided. We have evaluated our approach both in a simulation study and in network of real distributed smart cameras.

A visual sensor network (VSN) consists of resource-limited camera nodes which process the captured image data locally and collaborate with other cameras over a wireless network. The configuration of the network is a very important task in VSNs in order to adapt the available resources to the current requirements of the application. We focus on coverage and task assignment as a key configuration problem for VSNs. Due to rapid changes in the VSN's environment, a dynamic and online reconfiguration is highly needed. In this paper we introduce and evaluate a fast and distributed resource-aware reconfiguration algorithm. Our approach is based on simple optimization primitives to find good approximations and to keep the required data transfer in the VSN small. We analyze the communication complexity and compare our algorithm with a centralized configuration approach based on several scenarios with different complexity. Our experiments show that we can achieve the same configuration quality as with the centralized approach in almost all cases.

This paper presents a novel method for video-based traffic state detection on motorways performed on smart cameras. Camera calibration parameters are obtained from the known length of lane markings. Mean traffic speed is estimated from Kanade-Lucas-Tomasi (KLT) optical flow method using a robust outlier detection. Traffic density is estimated using a robust statistical counting method. Our method has been implemented on an embedded smart camera and evaluated under different road and illumination conditions. It achieves a detection rate of more than 95% for stationary traffic.

This paper presents a new method for estimating the level of service (LOS) on motorways in the compressed video domain. The method performs statistical computations on motion vectors of MPEG4 encoded video streams within a predefined region of interest to determine a set of four motion features describing the speed and density of the traffic stream. These features are fed into a Gaussian radial basis function network to classify the corresponding LOS. To improve the classification results, vectors of moving objects are clustered and outliers are eliminated. The proposed method is designed to be executed on a server system, where a large number of camera live streams can be analyzed in parallel in real-time. Evaluations with a comprehensive set of real-world training and test data from an Austrian motorway have shown an average accuracy of 86.7% on the test data set for classifying all four LOS levels. With a mean execution time of 48 microseconds per frame on a common server, hundreds of video streams can be analyzed in real-time.

Multimedia sensor networks [1] and visual sensor networks (VSN) [3] have been increasingly studied in recent years. However, the aspect of resource-awareness has just recently moved into the focus of research interests. Especially energy-aware systems that may be deployed in areas without fixed infrastructure have only recently achieved attention.

Summary form only given. The invited talks are: How to Compare Alternative Architectures by Radia Perlman of Intel; Portals 4: Enabling Application/Architecture Co-Design for High-Performance Interconnects by Ron Brightwell of Sandia National Laboratories; and Electronic-Photonic Integration within Switches and Routers by Mike Watts of MIT. Brief author biographies are also included.

This paper presents a real-time complex event detection concept for resource-limited multimedia sensor networks. A comprehensive solution based on Answer Set Programming (ASP) is developed. We show that ASP is an appropriate solution to detect a large number of simple and complex events (video-audio understanding) on platforms with limited resources e.g. power consumption, memory and processing power. We underline the major problems of the existing paradigms for complex event detection (based on e.g. logic programming and SemanticWeb), with a special focus on the major challenges which reduce the performance of real-time event detection. Finally, we demonstrate the high performance of ASP compared to that of Semantic Web.

This paper presents a smart resource-aware multimedia sensor network. We illustrate a surveillance system which supports human operators, by automatically detecting the complex events and giving the possibility to recall the detected events and searching them in an intelligent search engine. Four subsystems have been implemented, the tracking and detection system, the network configuration system, the reasoning system and an advanced archiving system in an annotated multimedia database.

Summary form only given. Automated code analysis is technology aimed at locating, describing and repairing areas of weakness in code. Code weaknesses range from security vulnerabilities, logic errors, concurrency violations, to improper resource usage, violations of architectures or coding guidelines. Common to all code analysis techniques is that they build abstractions of code and then check those abstractions for properties of interest. For instance a type checker computes how types are used, abstract interpreters and symbolic evaluators check how values flow, model checkers analyze how state evolves. Building modern program analysis tools thus requires a multi-pronged approach to find a variety of weaknesses. In this talk I will discuss and compare several program analysis tools, which MSR build during the last ten years. They include theorem provers, program verifiers, bug finders, malware scanners, and test case generators. I will describe the need for their development, their innovation, and application. Many of these tools had considerable impact on Microsoft's development practices, as well as on the research community. Some of them are being shipped in products such as the Static Driver Verifier or as part of Visual Studio. Performing program analysis as part of quality assurance is meanwhile standard practice in many software development companies. However several challenges have not yet been resolved. Thus, I will conclude with a set of open challenges in program analysis which hopefully triggers new aspiring directions in our joint quest of delivering predictable software that is free from defect and vulnerabilities.

A visual sensor network (VSN) consists of a large amount of camera nodes which are able to process the captured image data locally and to extract the relevant information. The tight resource limitations in these networks of embedded sensors and processors represent a major challenge for the application development. In this paper, we focus on finding optimal VSN configurations which are basically given by: 1) the selection of cameras to sufficiently monitor the area of interest; 2) the setting of the cameras' frame rate and resolution to fulfill the quality of service requirements; and 3) the assignment of processing tasks to cameras to achieve all required monitoring activities. We formally specify this configuration problem and describe an efficient approximation method based on an evolutionary algorithm. We analyze our approximation method on three different scenarios and compare the predicted results with measurements on real implementations on a VSN platform. We finally combine our approximation method with an expectation-maximization algorithm for optimizing the coverage and resource allocation in VSN with pan-tilt-zoom camera nodes.

Security and privacy protection are critical issues for public acceptance of camera networks. Smart cameras, with onboard image processing, can be used to identify and remove privacy sensitive image regions. Existing approaches, however, only address isolated aspects without considering the integration with established security technologies and the underlying platform. This work tries to fill this gap and presents TrustCAM, a security-enhanced smart camera. Based on Trusted Computing, we realize integrity protection, authenticity and confidentiality of image data. Multiple levels of privacy protection, together with access control, are supported. Impact on overall system performance is evaluated on a real prototype implementation.

In this paper, we introduce a fully autonomous vehicle classification system that continuously learns from largeamounts of unlabeled data. For that purpose, we proposea novel on-line co-training method based on visual and acoustic information. Our system does not need complicated microphone arrays or video calibration and automatically adapts to specific traffic scenes. These specialized detectors are more accurate and more compact than general classifiers, which allows for light-weight usage in low-cost and portable embedded systems. Hence, we implemented our system on an off-the-shelf embedded platform. In the experimental part, we show that the proposed method is able to cover the desired task and outperforms single-cue systems. Furthermore, our co-training framework minimizes the labeling effort without degrading the overall system performance.

Unmanned aerial vehicles (UAVs) have been recently deployed in various civilian applications such as environmental monitoring, aerial imaging or surveillance. Small-scale UAVs are of special interest for first responders since they can rather easily provide bird's eye view images of disaster areas. In this paper we present a hybrid approach to mosaick an overview image of the area of interest given a set of individual images captured by UAVs flying at low altitude. Our approach combines metadata-based and imagebased stitching methods in order to overcome the challenges of low-altitude, small-scale UAV deployment such as nonnadir view, inaccurate sensor data, non-planar ground surfaces and limited computing and communication resources. For the generation of the overview image we preserve georeferencing as much as possible, since this is an important requirement for disaster management applications. Our mosaicking method has been implemented on our UAV system and evaluated based on a quality metric.

Vehicle classification is an important task for various traffic monitoring applications. This paper investigates the capabilities of acoustic feature generation for vehicle classification. Six temporal and spectral features are extracted from the audio recordings. Six different classification algorithms are compared using the extracted features. We focus on a single sensor setting to keep the computational effort low and evaluate its classification accuracy and real-time performance. The experimental evaluation is performed on our embedded platform using recorded data of about 150 vehicles. The results are applied in our ongoing research on fusing video, laser and acoustic data for real-time traffic monitoring.

Smart cameras are embedded systems that perform on-board video content analysis and only report detected events instead of permanently streaming videos. Visual sensor networks aim at integrating smart cameras with wireless sensor networks. Camera sensors have higher requirements regarding computing power and communication bandwidth than those typically used in wireless sensor network applications. Consequently, power management is an even more critical issue. This work in progress presents an attempt to address this by combining high and low power radios as well as high and low performance computing systems in one single platform. This allows to control power consumption by selectively enabling only required components.

Distributed smart cameras (DSC) are an emerging technology for a broad range of important applications including smart rooms, surveillance, entertainment, tracking, and motion analysis. By having access to many views and through cooperation among the individual cameras, these DSCs have the potential to realize many more complex and challenging applications than single-camera systems. This article focuses on the system-level software required for efficient streaming applications on single smart cameras as well as on networks of DSCs. Embedded platforms with limited resources do not provide middleware services well known on general-purpose platforms. Our software framework supports transparent intra- and interprocessor communication while keeping the memory and computation overhead very low. The software framework is based on a publisher--subscriber architecture and provides mechanisms for dynamically loading and unloading software components as well as for graceful degradation in case of software- and hardware-related faults. The software framework has been completely implemented and tested on our embedded smart cameras consisting of an ARM-based network processor and several digital signal processors. Two case studies demonstrate the feasibility of our approach.

Distributed smart cameras (DSCs) are real-time distributed embedded systems that perform computer vision using multiple cameras. This new approach has emerged thanks to a confluence of simultaneous advances in four key disciplines: computer vision, image sensors, embedded computing, and sensor networks. Processing images in a network of distributed smart cameras introduces several complications. However, we believe that the problems DSCs solve are much more important than the challenges of designing and building a distributed video system. We argue that distributed smart cameras represent key components for future embedded computer vision systems and that smart cameras will become an enabling technology for many new applications. We summarize smart camera technology and applications, discuss current trends, and identify important research challenges.

In this article we present our software framework for embedded online data fusion, called I-SENSE. We discuss the fusion model and the decision modeling approach using support vector machines. Due to the system complexity and the genetic approach a data oriented model is introduced. The main focus of the article is targeted at our techniques for extracting features of acoustic-and visual-data. Experimental results of our "traffic surveillance" case study demonstrate the feasibility of our multi-level data fusion approach.

No abstract available.

Two trends emerge in recent image processing research: distributed computing and embedded processing. Both trends are exemplified in smart cameras which combine image sensing, image processing and communication on a single embedded device. Networks of distributed smart cameras help to overcome some hard problems that are inherent in single-camera systems. Designing, implementing and deploying image processing applications for cooperating distributed cameras is much more complex than for single-camera systems. In this paper, we focus on software services required for distributed embedded image processing on a network of smart cameras. We identify important middleware services and present our lightweight middleware implemented on our distributed SmartCams. A multi-camera tracking application demonstrates the benefits of our approach.

In this paper we describe a task allocation method, that utilizes genetic programming to find a suitable solution in an adequate time for this NP-complete combinatorial optimization problem. The underlying distributed embedded system is heterogenous, consisting of different processors with different properties such as core type, clock frequency, available memory, and I/O interfaces, interconnected with different communication media. In our applications, which are described as dataflow graphs, the number of tasks to be placed is much larger than the number of processors available. We highlight the difficulties when applying genetic programming to this problem and present our solutions and enhancements, accompanied with some simulation results.

Intelligent video surveillance (IVS) systems are based on the recent development of so called embedded smart cameras. Delivering a good service quality in IVS usually results in a higher level of computing activity and therefore in increased power consumption. This paper presents PoQoS, a novel approach that aims in maximizing the service quality (i.e., the number of IVS-services and their QoS) while minimizing the system's power consumption. PoQoS enables power-aware reconfiguration of services and hardware resources in distributed logical clusters of embedded smart cameras. In order to find optimal camera configurations during operation, PoQoS integrates PoSeGA, an online genetic multi-criterion optimization algorithm. A configuration manager properly selects among optimized camera configurations and consequently initializes intra-camera or intra-cluster power-aware reconfiguration with respect to application- and situation-specific context. The evaluation of PoQoS on the PoQoCam, a power-efficient embedded smart camera platform, shows the feasibility of the presented approach.

This paper reports on the integration of multi-camera tracking into an agent-based framework, which features autonomous task allocation for smart cameras targeting traffic surveillance. Since our target platforms are distributed embedded systems with limited resources, the trackers may only be active, if the target is in the camera's field of view. Consequently, the tracking algorithm has to migrate from camera to camera in order to follow the target, whereas the decision when and whereto the migration takes place is reached autonomously by the tracker. Consequently, no central control host is required. We further present different strategies on when to migrate a tracker, and how to determine the camera which observes the tracked object subsequently. We have realized the tracker's control by using heterogeneous mobile agents, which employ a state-of-the-art tracking algorithm for tracking. The tracking system has been implemented on our smart cameras (SmartCam) which are comprised of a network processor and several digital signal processors (DSPs) and provide a complex software framework.

This paper presents a dynamic task allocation method for smart cameras targeting traffic surveillance. Since our target platforms are distributed embedded systems with limited resources, the task allocation has to be light-weight, flexible as well as scalable and has to support real-time requirements. Therefore, surveillance tasks are not allocated to smart cameras directly, but to groups of smart cameras, so called surveillance clusters. We formulate the allocation problem as a distributed constraint satisfaction problem (DCSP) and present a distributed method for finding feasible allocations. Finally, a cost function is used to determine the optimal allocation of tasks. We have realized this dynamic task allocation using heterogeneous, mobile agents which utilize their agencies and our embedded software framework to find the most appropriate mapping of tasks in a distributed manner. The dynamic task allocation has been implemented on our smart cameras (SmartCam) which are comprised of a network processor and several digital signal processors (DSPs) and provide a complex software framework.

In next generation video surveillance systems there is a trend towards embedded solutions. Digital signal processors (DSP) are often used to provide the necessary computing power. The limited resources impose significant challenges for software development. Resource constraints must be met while facing increasing application complexity and pressing time-to-market demands. Recent advances in synthesis tools for Simulink suggest a high-level approach to algorithm implementation for embedded DSP systems. The model-based visual development process of Simulink facilitates simulation as well as synthesis of target specific code. In this work the modeling and code generation capabilities of Simulink are evaluated with respect to video analysis algorithms. Different models of a motion detection algorithm are used to synthesize code. The generated code targeted at a Texas Instruments TMS320C6416 DSP is compared to a hand-optimized reference. Experiments show that an ad hoc approach to synthesize complex image processing algorithms hardly yields optimal code for DSPs. However, several optimizations can be applied to improve performance.

A smart camera combines video sensing, high-level video processing and communication within a single embedded device. Such cameras are key components in novel surveillance systems. This paper reports on a prototyping development of a smart camera for traffic surveillance. We present its scalable architecture comprised of a CMOS sensor, digital signal processors (DSP), and a network processor. We further discuss the mapping of high-level video processing algorithms to embedded DSP-based platforms and identify typical pitfalls for the porting of software from desktops to embedded platforms. Our mapping strategies are demonstrated on an algorithm for automatic detection of stationary vehicles. This algorithm is migrated from a Matlab-based prototyping implementation to an embedded DSP implementation in our smart camera. Our implemented smart camera prototype streams the video data over an IP-network to a central monitoring station and is able to detect stationary vehicles and blocking cargo on highways within the required real-time constraints of six seconds.

Model-based monitoring determines faults in a supervised system by comparing the available system's measurements with a priori information represented by the system's mathematical model. Especially in technical environments, a monitoring system must be able to reason with incomplete knowledge about the supervised system, to process noisy and erroneous observations and to react within a limited time. We present MOSES, a model-based monitoring system which is based on imprecise models where the structure is known and the parameters may be imprecisely specified by numerical intervals. As a consequence, only bounds on the trajectories can be derived with imprecise models. These bounds are computed using traditional numerical integration techniques starting from individual points on the external surface of the model's uncertainty space. When new measurements from the supervised system become available, MOSES checks the consistency of this new information with the model's prediction and refutes inconsistent parts from the uncertainty space of the model. A fault in the supervised system is detected when the complete model's uncertainty space has been refuted. MOSES bridges and extends methodologies from the FDI and DX communities by refining the model's uncertainty space conservatively through refutation, by applying standard numerical techniques for deriving the trajectories of imprecise models and by exploiting the measurements as soon as possible for online monitoring. The performance of MOSES is evaluated based on examples and by online monitoring a complex heating system.

The development of parallel applications is tedious and more complex than a single-processor solution. We have developed PEPSY, a prototyping environment for multi-DSP systems, with the primary goal to automate the design and implementation of parallel DSP applications. Given an extended data flow graph of the DSP application and a description of the target multiprocessor system, PEPSY automatically maps and schedules the DSP application onto the multiprocessor system and generates complete code for each processor. PEPSY excels in an accurate performance estimation. The design goals of the parallel application can, therefore, be verified prior to its implementation. With PEPSY, parallelization of a DSP application onto various processors can be realized within minutes.

We present a novel approach to model inter-processor communication in multi-DSP systems. In most multi-DSP systems, inter-processor communication is realized by transferring data over point-to-point links with hardware FIFO buffers. Direct memory access (DMA) is additionally used to concurrently transfer data to the FIFO buffers and perform computation. Our model accounts for the limited size of the communication buffers as well as concurrent DMA transfer. This novel communication model is applied in our rapid prototyping environment for optimizing multi-DSP systems. Given an extended data flow graph of the DSP application and a description of the target multi-processor system our rapid prototyping environment automatically maps the DSP application onto the multi-processor system and generates a schedule for each processor.