The reliability of microelectromechanical systems (MEMS) becomes increasingly important due to the fast market growth, especially in medical, automotive, and aerospace applications, where high reliability performance under harsh environmental conditions is crucial. The aim of this work is to test the behavior and performance of two-dimensional (2D) resonant MEMS mirrors, which are driven electrostatically, under different temperatures. Therefore, six mirrors and their driver application specific integrated circuits (ASIC) are tested inside a temperature chamber, where the temperature is swept between -40°C and 90°C multiple times and different mirror parameters are logged for both axes of the mirrors and the results are evaluated afterwards. Especially the driving frequencies of the axes are monitored. The results show that there is a nearly linear inverse relation between driving frequency of the mirror's fast axis and temperature, as the frequency rises with a decreasing temperature, while for the slow axis the relation between frequency and temperature is more complex. Moreover, the operation of the mirror is stable over the tested temperature range, as no failures are observed.

ShapeFuture will drive innovation in fundamental Electronic Components and Systems (ECS) that are essential for robust, powerful, fail-operational and integrated perception, cognition, AI-enabled decision making, resilient automation and computing, as well as communications, for highly automated vehicles. The overarching vision of ShapeFuture is to bring ECS Innovation to the heart of Europe’s Mobility Transformation, thereby elevating sovereignty by perfecting programmable ECS solutions for intelligent, safe, connected, and highly automated vehicles. In this paper, we detail not only the vision and mission of the ShapeFuture project, but we also showcase the results achieved during the first year.

Manufacturing advances in the field of micro-electro-mechanical systems (MEMS) enabled the realization of MEMS mirrors. These MEMS mirrors, which are as small as a few square millimeter, are promising candidates for various types of automotive applications requiring laser beam steering. Prime application examples are Head-Up Displays (HUD) or Light Detection and Ranging (LiDAR) sensors. In this work, we present the latest advancements and concepts for sensing, actuating, and controlling oscillating comb-drive based MEMS mirrors. Also an efficient approach towards high-voltage generation and actuation is presented which is capable to reduce the MEMS mirror's power dissipation by factors. Furthermore, the performance of the presented control system is depicted by means of an automotive LiDAR use-case which was integrated into an automated demo vehicle.

Autonomous driving is disrupting the automotive industry as we know it today. For this, fail-operational behavior is essential in the sense, plan, and act stages of the automation chain in order to handle safety-critical situations on its own, which currently is not reached with state-of-the-art approaches.The European ECSEL research project PRYSTINE realizes Fail-operational Urban Surround perceptION (FUSION) based on robust Radar and LiDAR sensor fusion and control functions in order to enable safe automated driving in urban and rural environments. This paper showcases some of the key exploitable results (e.g., novel Radar sensors, innovative embedded control and E/E architectures, pioneering sensor fusion approaches, AI-controlled vehicle demonstrators) achieved until its final year 3.

In advanced driver-assistance systems, LiDAR data are used for range detection and obstacle avoidance in combination with other sensors. The frame rate of a LiDAR sensor corresponds to the data availability that is crucial for efficient data fusion. In 1D micro-scanning LiDAR, pixel accumulation is introduced to increase data signal-to-noise ratio and typically performed a fixed number of times that directly affects pixel acquisition time and frame rate. In this paper, we present an adaptive pixel accumulation algorithm that not only reduces required on-chip memory array by compressing LiDAR raw data, but also increases data availability for occupancy grid computation by enabling an early peak detection and eliminating unnecessary accumulation cycles whenever possible. We implemented this concept on FPGA and compared its efficiency with a state-of-the-art approach. Presented simulation and measurement results show an improvement of data availability in short and mid-range scenarios or when detecting a highly reflective target.

Computer vision algorithms are influenced by variations in lighting conditions. Images, independent of lighting conditions have the potential to improve tasks such as material and object classification. For Time-of-Flight (ToF) cameras the largest variation in lighting condition is caused by the distance between the camera and the object. ToF cameras are intended for 3D distance measurement. However, the features which enable distance measurement can be used to realize methods to record distance normalised grey images.In this paper, we explore methods to extract direct reflectance estimates from ToF measurements. We propose two novel methods relying on coded modulation (CM) and compare them to a method that can convert data from the state-of-the-art continuous wave (CW) measurement method. With the invention of the CM based methods, we can realize the normalization in a single frame measurement, compared to the four frames recorded by the CW method.All three methods are evaluated based on simulation results and in-laboratory measurements. We are able to demonstrate that our novel methods, relying on CM, can achieve the desired measurement behaviour.

Automated driving has been expected for decades. The first systems, which at least partially automate the vehicle, have been installed in higher priced vehicles for several years. In the near future, however, many more competencies are to be transferred to the systems and the vehicle will thus be fully automated. Such systems receive their data from various sensor systems such as Light Detection And Ranging (LiDAR). Therefore, it is essential that this information is transmitted correctly and reliably to the environmental perception system. In order to ensure this, redundancy of sensors is a key factor in addition to diversity. For example, multiple, independently controlled MEMS-based LiDAR systems can be operated synchronously. This requires the selection of a Master system which can be reliably followed by all Slave systems. In this publication, an architecture for MEMS-based Micro-Scanning LiDAR systems is proposed to determine the appropriate system as Master. The architecture has been implemented in an FPGA prototyping platform to demonstrate its feasibility and evaluate its performance.

Autonomous driving has the potential to disruptively change the automotive industry as we know it today. For this, fail-operational behavior is essential in the sense, plan, and act stages of the automation chain in order to handle safety-critical situations by its own, which currently is not reached with state-of-the-art approaches.The European ECSEL research project PRYSTINE realizes Fail-operational Urban Surround perceptION (FUSION) based on robust Radar and LiDAR sensor fusion and control functions in order to enable safe automated driving in urban and rural environments. This paper showcases some of the key results (e.g., novel Radar sensors, innovative embedded control and E/E architectures, pioneering sensor fusion approaches, AI controlled vehicle demonstrators) achieved until year 2.

A continuously growing interest towards autonomous vehicles highlights the need of robust sensors that will reliably operate under harsh environmental conditions. In this paper, we analyze environmental disturbances that influence the optical sensing accuracy of a MEMS-based LiDAR (Light Detection and Ranging) sensor, and propose an adaptive control scheme of the MEMS mirror that lower the impact of disturbances on the sensing accuracy. This scheme exploits data from various internal and external monitors and adapts MEMS mirror control parameters such that the angular RMS jitter does not exceed 15m° when environmental conditions change. Measurement results show that this approach not only introduces robustness in the MEMS mirror control loop but also improves overall reliability of ADAS applications.

Time of Flight (ToF) based three-dimensional (3D) imaging sensors, such as Light Detection and Ranging (LiDAR) sensors or ToF cameras, can be used to depict their surroundings in form of a point cloud. The ToF method measures the time an emitted light signal takes to be reflected by a point in space and derives the distance from the known travel time of light. More and more ToF based 3D sensors are developed for automotive use as their target application. Automotive sensors are part of a safety critical application. Therefore, it is important to ensure that the provided sensor data is accurate with a known probability. In this paper, we describe and evaluate test procedures for quantitative and qualitative performance evaluation of ToF/LiDAR sensors with focus on automotive use. We propose a LiDAR error and influence model from which we derived eight test areas. We described and conducted tests for six of the eight test areas. The described test cases were evaluated on three LiDAR sensors and one ToF camera. The results show that targets and test procedures need to be adapted to the specific tested ToF/LiDAR sensor. Especially noticeable where influences on test procedures due sparse sensor resolution. Furthermore, the test results show that target application specific tests can provide additional information on the behaviour of the sensor.

Synchronized and precisely timestamped data from perception sensors is highly advantageous for the low-level fusion of multiple sensor data. Many open-available, low-cost perception sensors do neither provide hardware support for precise clock synchronization, nor provide timestamps with their measurement data. In this work, we present an approach to enable synchronization and accurate timestamping of hardware-triggerable sensors in multi-sensor perception systems.We utilize a hybrid timestamping approach, taking into account the timestamp of a hardware trigger and the software timestamp. The presented timestamping approach utilizes the trigger time to assign precise timestamps to the data streams of the perception sensors. Precise timestamps are mandatory in order to achieve a high perception performance in dynamic applications which utilize low-level data streams.Additionally, we present an implementation of the approach on a multi-sensor perception platform, archiving a timestamp precision in the range of 2 ms. An existing Robot Operating System (ROS) architecture of the platform is extended to assign hybrid timestamps to the data streams. Additionally, we present a pedestrian detection implementation which fuses the timestamped data into a representation.

Advancing the current Advanced Driver Assistance Systems (ADAS) is coupled with introducing novel technologies into the automotive domain such as Light Detection and Ranging (LiDAR). LiDAR is attributed as a key-technology that will be one of the key enablers for safe and reliable automated driving. Considering the fact that vehicles nowadays rely on the driver in safety-critical situations leads to the problem that in a fully-automated driving scenario the vehicle needs to control every possible situation on its own. This increases the requirements and the overall safety level of the system but also for each component and needs a gradual transition from fail-safe to fail-operational behavior at least as long as the occupants and other road participants could be endangered.This publication introduces a novel system architecture of a fail-operational 3D Flash LiDAR System that enables dynamic system degradation during run-time as well as internal built-in self-test (BIST) for automated failure injection tests. The novel fail-operational system architecture is able to handle critical temperature ranges as well as long-term memory faults.

In recent years, topics like autonomous driving increased the demand on robust environmental sensors. Depth sensors are most commonly used. Solid state Light Detection And Ranging (LiDAR) sensors are well suited for these applications. The measurement principle is based on measuring the phase and consequently the delay of emitted and reflected light. Problems arise if strong reflectors, like street-signs, impair the measurement. In this paper, we present a novel algorithm for depth calculation, based on indirect Time-of-Flight (ToF) data. With this approach it is possible to separate multiple reflectors in the scenery. This allows the generation of multiple depth images. In our approach an arbitrary number of different code sequences are applied as modulation signal. With these code sequences we generate a so called ToF-matrix. With this ToFmatrix, the measured environmental response can be mapped to a distance. As our evaluation shows, our method is able to achieve results with more information compared to conventional ToF-imaging. We demonstrate the separation of the reflection of a street-sign, from a target. This algorithm enables the usage of indirect ToF in automotive areas. We believe that this versatile calculation approach can increase the benefit of indirect LiDAR application for autonomous driving.

In intelligent vehicles, it is indispensable to have reliable Advanced Driver-Assistance Systems (ADAS) on board. These ADAS require various types of sensors, like Light Detection and Ranging (LiDAR). Nowadays, drivers delegate some responsibilities to their highly automated vehicles; however, it is not legally secured. Nevertheless, the legislator will, in the future, deal with automated vehicles. The fundamentals will be laid to ensure that the transfer of responsibilities will be permitted under certain conditions. Car manufacturers, on the other hand, must ensure that components are safe and reliable. With LiDAR, this could be achieved with Micro-Electro-Mechanical System (MEMS) technology. As with humans as drivers, it is also advantageous for intelligent systems if obstacles in the environment are detected promptly. Especially when the obstacles are moving, it helps to initiate appropriate measures, such as braking. Therefore, it is attempted to extend the Field-of-View (FoV) of the various sensors. By synchronising multiple MEMS mirrors, it is able to extend the FoV of the LiDAR part in an environmental perception system. In this publication, an architecture is proposed for MEMS-based Micro-Scanning LiDAR Systems to achieve synchronisation of multiple independently controlled MEMS mirrors. The architecture was implemented in an FPGA prototyping platform to show its feasibility and evaluate its performance.

In recent years, application fields such as industrial automation and indoor robot navigation increased the demand on reliable localization systems. Simultaneous mapping and localization systems often depend on depth imaging in order to reconstruct the scene. Time-of-Flight sensors prove to be well suited for these applications, however are impaired by different error sources. The measurement principle is based on measuring the phase and consequently the delay of emitted and reflected light. Specular surfaces can cause pixel saturation, while the periodicity of the measured phase leads to ambiguous distances. In this paper, we aim to solve these problems by proposing a new Time-of-Flight depth sensing approach. By combining the emerging coded modulation method with traditional depth sensing, we are able to unify the advantages of both methods. Images captured with coded modulation show a pixel response only within selected distance limits. In contrast traditional continuous wave Time-of-Flight imaging exhibits a superior signal-to-noise ratio. This method enables to mask erroneous distance measurements, allowing Time-of-Flight sensors to produce more reliable depth measurements and gain traction in the industrial environment. As our evaluation shows, our method is able to remove the influence of specular surfaces, and is capable of masking ambiguous distance measurements. Furthermore, our approach improves the system behavior by enabling more robust exposure time control.

Among the actual trends that will affect society in the coming years, autonomous driving stands out as having the potential to disruptively change the automotive industry as we know it today. For this, fail-operational behavior is essential in the sense, plan, and act stages of the automation chain in order to handle safety-critical situations by its own, which currently is not reached with state-of-the-art approaches also due to missing reliable environment perception and sensor fusion. PRYSTINE will realize Fail-operational Urban Surround perceptION (FUSION) which is based on robust Radar and LiDAR sensor fusion and control functions in order to enable safe automated driving in urban and rural environments. In this paper, we detail the vision of the PRYSTINE project and we showcase the results achieved during the first year.

In recent years, application fields such as secure face recognition or autonomous driving increased the demand on efficient depth sensing systems. Time-of-Flight (ToF) sensors are well suited for these applications. The measurement principle is based on measuring the phase and consequently the delay of emitted and reflected light. For this delay measurement a continuous wave signal is emitted. Coded modulation replaces this continuous wave signal with code sequences. This enables new possibilities as the measurement range of the camera is adjustable with coded modulation. A well suited way for the characterization of this modulation method is a simulation framework. In this paper, we present a simulation framework for coded modulation ToF imagers. We present a detailed description of our PMD technology. From this theoretical description, we adapt an existing simulation model for coded modulation ToF cameras. The model of the camera considers various different noise sources. Furthermore depth calculation principles of coded modulation are introduced. As our evaluation shows, our framework is able to simulate real life behavior of coded modulation. Furthermore we are able to model the correlation form, and consequently the depth and intensity measurement behavior. In the end we evaluate our simulation results with real live measurement data. With this framework easy to use evaluation of coded modulation will enable new applications for this technique.

We present an approach to fuse radar and time-of-flight (ToF) range sensor data into an occupancy grid. Fusing the low-level data at sensor level prevents the loss of precious information during compression and pre-processing. Constructing the low-level occupancy grid from raw sensor data enables the detection of occupied cells which are not clearly visible by any of the single sensors. Fusion of the heterogeneous sensor data enhances the perception quality since single sensors fail in certain conditions. Thus, the fusion at low-level holds a high potential to enhance the perception quality for automotive/robotic applications. We demonstrate our approach with real-world data from a mobile sensor platform with three ToF cameras and a 77 GHz high-resolution radar sensor. An occupancy grid is created whenever synchronized sensor data from all sensors is available. The proposed method performed successful detection of multiple pedestrians in different test scenarios. Our approach to build an occupancy grid from radar and optical range sensors can be used as a base in various short-range perception applications (e.g., in robotics or mobile devices).

Autonomously driving vehicles require higher safety and reliability standards than traditional human-driven vehicles as they need to be able to handle safety-critical situations on their own. Therefore, these systems needs to demonstrate fail-operational behavior to ensure safety of the passengers by basic car controls. Especially silent failures of semiconductor devices can be critical from a safety point of view. Semiconductor devices fail abruptly and cannot be detected in advance. This paper presents a novel sensor approach to detect those kind of silent failures ahead of time and to ensure safety for future advanced driver-assistance systems (ADAS) such as LiDAR (Light Detection and Ranging). We have evaluated the design of our novel sensor concept in SystemC which will be implemented in a LiDAR system to mitigate silent failures as well as enable dynamic safety contracts.

External sensing for automative applications are key tools for the development of Advanced Driver Assistance Systems (ADAS), since they can sense and analyse the environment around the vehicle by providing pictures of the scene behind the vehicle. Parking assistance systems are already available in the market. However, most of these applications are based on ultrasonic sensors, wide-angle image cameras, RADAR, etc, which present some drawbacks such as dependency to light conditions or high maintenance cost, among others. This paper proposes an approach for assisting drivers to park through the processing of data derived by a 3D Time-of-Flight (ToF) camera and the reconstruction of the objects identified around the vehicle. The proposed technique is focused on fusion of two parallel processing technologies, a visual one through the intensity image and a spatial one through the point cloud. Both of them are centered on the detection of the vehicle's plate to estimate its position and determine free spots in the parking. This novel methodology improves the detection of surrounding elements, since it helps to solve two main problems with this kind of devices: 1) the degraded performance under bright ambient light problem (occurring mainly in outdoor parkings), that causes shadows and brightness in the images, hindering its process to detect objects; and 2) the limited detection of low reflection objects such as dark cars. Moreover, this fusion allows to link each pixel of the image with a 3D position, and vice versa, giving the point cloud a visual reference. The system is evaluated through a Renault Twizy platform in real conditions.

Among the actual trends that will affect society in the coming years, autonomous driving stands out as having the potential to disruptively change the automotive industry as we know it today. As a consequence, this will also highly impact the semiconductor industry and open new market opportunities, since semiconductors play an indispensable role as enablers for automated vehicles. Fully automated driving has been identified as one major enabler to master the Grand Societal Challenges of safe, clean, and efficient mobility. For this, fail-operational behavior is essential in the sense, plan, and act stages of the automation chain in order to handle safety-critical situations by its own, which currently is not reached with state-of-the-art approaches also due to missing reliable environment perception and sensor fusion. PRYSTINE, which was the highest ranked ECSEL project proposal in 2017, will realize Fail-operational Urban Surround perceptION (FUSION) which is based on robust Radar and LiDAR sensor fusion and control functions in order to enable safe automated driving in urban and rural environments. Furthermore, PRYSTINE will strengthen and extend traditional core competencies of the European industry, research organizations, and universities in smart mobility and in particular in the electronic component and systems and cyber-physical systems domain.

Supporting visually impaired people in their everyday's life is of crucial importance. Thanks to the technological advancements in semiconductors and cyber-physical systems, novel powerful tools and capabilities emerged recently. The integration of miniaturized Time-of-Flight 3D imaging modules combined with vast CPU and GPU power into commercially available smart phones, is a major enabler for a future virtual white cane aiding visually impaired people. This work presents a virtual white cane prototype which in particular targets visually impaired persons and vulnerable road users in general. It exploits the unique 3D environment perception features of a commercially available smart phone in order to perceive the environment in a three dimensional way based on the Time-of-Flight 3D imaging technology. This work not only introduces new concepts, such as a combined v disparity / RANSAC ground plane detection and the so-called Conservative Polar Histogram, it also considerably outperforms the object detection performance compared to state-of-the-art approaches.

The heart beat is one of the basic vital signs, but the pulse wave can communicate much more information than the beat frequency of the heart. Through the heart rate variability (HRV) among other things, stress level and drowsiness can be inferred. Reliable HRV measurements are commonly obtained by electrocardiography (ECG). In this paper we analyse the correlation between HRV and pulse rate variability (PRV) obtained from contact or remote photoplethysmography (PPG) on a Time-of-Flight (ToF) camera from PMD Technologies AG. The ToF camera is independent of passive illumination as it uses an infrared (IR) light source, which is invisible to the human eye. Therefore, potential use cases include driver monitoring or convenient heart rate measurement with smart phones. We will demonstrate methods for using a ToF camera for contact and remote PPG. For contact PPG the sensor is mounted directly on the subjects body. The pulse wave is calculated as the mean amplitude intensity of all pixels. For remote PPG the sensor is mounted to measure the subjects face. The pulse wave cannot be found in every part of exposed skin. Therefore, the part of the skin with the clearest signal is localised and the pulse wave is recovered from there, by means of passband filtering and blind source separation. Our results show that contact PPG on a ToF camera has high accuracy and highly correlates with a commercial IR pulse oximetry device, as well as with an ECG grade chest belt. For remote PPG our method shows correlation with the ToF contact PPG signal.

We present an open hardware and software platform to efficiently fuse heterogeneous sensor data in an automotive/robotic context. The framework presented in this paper provides researchers a base platform in order to develop and evaluate sensor fusion strategies. In contrast to similar approaches, this framework exploits in particular the raw radar data and enables the fusion at low-level. The proposed system utilizes low-level data from radar sensors as well as indirect (e.g. 3D imaging) and direct (e.g. LIDAR) Time-of-Flight (ToF) sensors. After a configurable amount of pre-processing at sensor-level, the sensor data is transferred to a centralized platform and aligned temporally and spatially. We demonstrate the transformation of radar data into the 3D coordinate system in order to fuse it with point cloud data from ToF sensors. Due to the modular structure of the framework, it also enables the exploration of various system partitioning concepts.

Time-of-Flight 3D imaging, using the indirect measuring method that employs photonic mixing devices, increases in popularity. This is due to the recent availability of accurate and miniaturized Time-of-Flight cameras that can be integrated into small embedded devices. However, providing a system comprising a camera and hardware-accelerated processing, which is useable for various application types with diametrically opposed use-case requirements, is not trivial. Here we introduce a fast and flexible hardware/software co-processing framework for Time-of-Flight 3D imaging, which can be used for various kinds of use-case applications. This is achieved in particular by providing line-grained hardware-acceleration for Time-of-Flight algorithms and computer vision applications. Furthermore, this hardware/software framework enables and features state-of-the-art multi-core, mixed-criticality, and real-time concepts, which is essential for critical applications in the automotive and industrial domains. We demonstrate the feasible implementation and integration of the framework by means of a mixed-critical automotive system demonstrator. This demonstrator enables various types of use-case scenarios, such as gesture recognition or indoor navigation, by employing the AURIX automotive processor from Inlineon Technologies and the Xilinx Zynq platform. For instance, if a typical gesture recognition use-case is addressed, the framework is capable to provide high-quality depth images and 3D-point cloud data with nearly 100 FPS and introduces an average calculation error of only 0.08 mm. Most importantly, a high level of flexibility in software and hardware is preserved in order to support other use-case scenarios at the same time.

Automated driving systems have to operate at the highest level of robustness and safety. Thus, redundancy and diversity of the deployed systems are inevitable in order to guarantee the functionality in any possible scenario. Today, the most used sensor technologies for environment perception are color cameras, radar, light detection and ranging (LIDAR), and ultrasonic sensors. This work evaluates the feasibility of a 3D Time-of-Flight (ToF) camera to be used as environmental perception sensor in robotics and automated/assisted driving. To examine the performance of the sensor in the field, a ToF processing platform is attached to a 1:5 scaled remote control vehicle. An algorithm, which detects and reacts to obstacles in real-time, is designed and implemented on an AURIX automotive safety-microcontroller as major part of a mixed-criticality application. The embedded system highly benefits from the low computational effort required by ToF imaging (in contrast to stereo vision). Utilizing all three cores of the AURIX, the system achieves frame-rates of up to 30 frames per second (FPS).

Industrie 4.0, Industrial IoT, and cyber-physical production systems in general introduce high levels of automation. Hence, at shop floor level, for example, the interfaces between human and machines are crucial. If a robot's environment perception is not robust and fail-safe, humans may not be detected properly, which may cause critical consequences. However, given the resource constraints of safety controllers typically used in critical application domains, the implementation of complex computer vision algorithms is often challenging. Here we explore the capabilities the indirect Time-of-Flight environment perception technology provides in order to speed up complex computer vision algorithms. In particular we are investigating the use-case of human detection in the domains of critical and resource-constrained applications, such as factory automation and automotive. We demonstrate that preprocessing based on Time-of-Flight 3D data can reduce computation time of typically used computer vision algorithms, such as Viola-Jones, by up to 50%. Furthermore, we showcase a human detection demonstrator case-study implemented on an AURIX processing system that represents a state-of-the-art safety controller. By exploiting the Time-of-Flight technology's depth and amplitude data in a clever way, safety-critical human detection is enabled on the AURIX platform. Compared to competing environment perception technologies, the outlined solution is hardly achievable with structured light or stereo visioning due to the safety controller's resource constraints.

Time-of-Flight 3D imaging systems are promising transceivers for image sensor based optical communication. 3D sensing based on Time-of-Flight is the most miniaturized depth imaging technology available and is currently being integrated into consumer electronics such as smart phones. Optical line-of-sight communication with a depth imaging system offers complete location-awareness of the communication partner. This enables new opportunities in fields like secure authentication, augmented reality or vehicle to vehicle communication. We show how Time-of-Flight systems are capable of using phase shift keying of pulsed light (PLPSK) to transmit data over a line-of-sight connection. PLPSK is still unexplored in the domain of image sensors and enables Time-of-Flight sensors to transmit multiple bits per frame at rates of over 7 kHz. A serious problem however are asynchronous modulation signals, causing frequent transmission errors and impairing proper communication. In this work, we formulate and discuss the severity of the problem and propose a synchronization procedure. We evaluate our solution with a prototype system, and show that it is possible to reach a synchronization success rate of nearly 100% over a distance of 9 meter.

In recent years, fields such as industrial automation, virtual and augmented reality and autonomous robotics increased the demand for location-awareness of electronic devices. Image sensor based inside-out localization and tracking systems are sufficiently accurate to determine the position and orientation of electronic devices. Without additional sensors however, these systems are impaired in reaching high update-rates, handling fast motions, and tend to be unable to provide localization with low latency. We present a new localization approach in our work, using Time-of-Flight 3D sensors in combination with small reflective markers. This allows to establish high-performance optical localization systems, delivering the position and orientation of a device at a rate of several hundred Hz. A novel Time-of-Flight 3D sensing procedure is introduced, enabling to measure the 3D positions of fast moving targets at unprecedented frame-rates. With this work, we aim to close the gap between indoor positioning and motion tracking, enabling a new class of location-aware devices.

RFID/NFC technology is widely spread nowadays and applications can be found in our everyday life, for example, in payment, transportation, logistics, healthcare, and access control. State-of-the-art contactless and passive authentication solutions implement relatively large coils outside of the chip. Therefore, the minimum size is in the order of a few square centimeters, which limits their use for tagging of certain small-sized goods. On top of that, those miniaturized solutions which are available today provide only limited security measures. Here we introduce miniaturized system-in-package contactless authentication devices. This novel solution integrates Infineon Technologies' CIPURSE™move IC, which is a state-of-the-art security solution featuring an open security standard, into embedded Wafer Level Ball Grid Array (eWLB) packages, together with HF-antennas, ferrites, as well as discrete elements that improve HF-coupling characteristics. The presented devices provide better HF-coupling characteristics than Coil-on-Chip approaches, which also enable verification of authenticity of tagged products through NFC-enabled smart phones. Thanks to the miniaturized package sizes of 3x3 mm, integration into high-priced products, casings, consumable materials, etc., can be achieved in a discreet way. Furthermore, the integrated CIPURSE™ chip enables not only the anticounterfeiting use-case, but also micropayment, ticketing, access control, and password storage in a secured way. Therefore, this miniaturized contactless authentication solution will open up whole new fields of applications.

Since April 2014 the Artemis/ECSEL project EMC2 is running and provides significant results. EMC2 stands for "Embedded Multi-Core Systems for Mixed Criticality Applications in Dynamic and Changeable Real-Time Environments". In this paper we report recent progress on technical work in the different workpackages and use cases. We highlight progress in the research on system architecture, design methodology, platform and operating systems, and in qualification and certification. Application cases in the fields of automotive, avionics, health care, and industry are presented exploiting the technical results achieved.

Computer vision is becoming more and more important in the fields of consumer electronics, cyber-physical systems, and automotive technology. Recognizing and classifying one's environment reliably is imperative for safety-critical applications, as they are omnipresent, e.g., in the automotive or aviation domain. For this purpose, the Time-of-Flight imaging technology is suitable, which enables robust and cost-efficient three-dimensional sensing of the environment. However, the resource limitations of safety- and security-certified processor systems as well as complying to safety standards, poses a challenge for the development and integration of complex Time-of-Flight-based applications. Here we present a Time-of-Flight system approach that focuses in particular on the automotive domain. This Time-of-Flight imaging approach is based on an automotive processing platform that complies to safety and security standards. By employing state-of-the-art hardware/software and multi-core concepts, a robust Time-of-Flight system solution is introduced that can be used in a mixed-critical application context. In this work we demonstrate the feasible implementation of the proposed hardware/software architecture by means of a prototype for the automotive domain. Raw Time-of-Flight sensor data is taken and 3D data is calculated with up to 80 FPS without the usage of dedicated hardware accelerators. In a next step, safety-critical automotive applications (e.g., parking assistance) can exploit this 3D data in a mixed-critical environment respecting the needs of the ISO 26262.

ECC is an asymmetric encryption providing a comparably high cryptographic strength in relation to the key sizes employed. This makes ECC attractive for resource-constrained systems. While pure hardware solutions usually offer a good performance and a low power consumption, they are inflexible and typically lead to a high area.

RFID-based and NFC-based applications can be found, apart from others, in security critical application fields, such as payment or access control. For this purpose, Elliptic-Curve Cryptography (ECC) is commonly used hardware integrated in resource constrained applications in order to provide authenticity and data integrity. On the one hand, specialized crypto hardware approaches provide good performance and consume low power. On the other hand, they often lack flexibility, caused, for example, by hardware integrated protocols and cryptographic parameters. Here we present a flexible and lightweight ECC-based authentication solution that takes into account resource constrained systems. This technique permits to shift parts of the computational intense ECC calculations from the resource constrained device to the authentication terminal. By employing a security controller with a small multi-purpose hardware acceleration core, high computation speed is achieved and a maximum level of flexibility is maintained at the same time. We demonstrate the feasible implementation of the proposed technique by means of an Android-based reader / smart card system, which represent a prime example of contemporary power-constrained and performance-constrained embedded systems. An ECC-based authentication can be carried out on average within 25 ms and checked against a back-end server within 66 ms in a secured manner. Thus, a secured and flexible one-way authentication system is given that shows high performance. This solution can be utilized in a wide variety of application fields, such as anti-counterfeiting, where flexibility and low chip prices are essential.

Developing an embedded system today means integrating a bundle of features into a constrained and complex system. Examples are Near Field Communication (NFC) handsets like smart phones, which will hit the 1.2billion mark in 2017. Model Driven Architecture (MDA) is an approach to handle this complexity. Challenges in MDA are the verification of power-requirements across the development phases and to find the suitable abstraction for the power models for higher abstraction levels. Therefore, we propose a framework for MDA to support cross-verification of these requirements. We implemented this framework and made a case study of developing a power-aware NFC-System. The case study shows that the framework allows a power-verification with an accuracy of 10%.

Testing hardware and software components regarding their fault detection and fault handling capabilities is of vital importance. However, considering the fact that security systems are built using several distributed hardware components (e.g., reader/smart card authentication system), testing each component individually is insufficient. Because novel system-wide multi-fault attack campaigns can be conducted, fault propagation as well as fault handling of the entire system must be regarded. State-of-the-art emulation-based fault analysis approaches neglect this system aspect as well as the fault impact on power dissipation and power supply. Here, we present a novel analysis methodology that characterizes the behavior of complete systems during the design phase, in terms of fault handling, power dissipation, and power supply. Emulation-based techniques are applied to provide cycle accurate analysis information of the system-under-test in real time. The presented approach is of importance when it comes to test resource constrained, dependable, and high secure system designs. We demonstrate the application of this approach by means of a reader/smart card authentication system. Furthermore, we show how system level-based multi-fault attacks can be emulated and how the resulting system behavior (e.g., power consumption, power supply, information leakage) can be exploited to extract security relevant information.

Power consumption and thermal behavior are important characteristics that need to be explored and evaluated during a product's development cycle. If not handled properly, the consequences are, for example, increased mean-time-to-failure and fatal timing variations of the critical path. In the field of contactlessly powered reader/smart card systems, a magnetic field strength exceeding the allowed maximum threshold may harm the smart card's hardware. Thus, secure smart cards must be designed to cope with faults provoked by power oversupply and thermal stress. Proper fault detection and fault handling are imperative tasks to protect internal secrets. However, state-of-the-art design exploration tools cover these smart card specific power and thermal stress issues only to some extent. Here we present an innovative high level simulation approach used for exploring and simulating secure reader/smart card systems, focusing on magnetic field oversupply and thermal stress evaluations. Gate-level-based power models are used besides RF-channel models, thermal models, and thermal effect models. Furthermore, fault injection techniques are featured to evaluate the fault resistance of a smart card system's software implementation. This framework grants software and hardware designers a novel opportunity to detect functional, power, thermal, and security issues during the design time. We demonstrate the usage of our exploration framework and show an innovative hardware design approach to prolong the lifetime of smart card electronics, which are exposed to high magnetic field strengths.

More than 500 million Near Field Communication (NFC) devices will be delivered in 2014. This technology enables a lot of application fields like using it for bridges to embedded systems (e.g., smart meters). With this wireless bridge the user can interact with the embedded system using an off-the-shelf NFC-enabled smart phone. The user of such a bridge also trusts in the system's security. Furthermore, this security should not lead to an excessive battery drain of the smart phone nor the embedded system. This publication deals with these concerns and shows a method called PtNBridge. The method secures the whole communication path from the smart phone application to the accessed module in the embedded system (e.g., power sensor of the smart meter). To take account of the energy consumption, the PtNBridge has been analyzed and optimized to avoid an excessive battery drain. Two variants of the PtNBridge have been implemented, which aim for two different goals of power-aware security.

Test and verification are essential parts during a product's development cycle. Simulation and emulation are well known techniques to test and verify the functionality of a design-under-test (DUT) before its tape-out. However, there are additional issues like peak power consumption and supply voltage drops, which can compromise a hardware's functionality. These issues are only partly covered by nowadays functional hardware emulation test and verification approaches. This paper presents a comprehensive emulation methodology. It combines functional hardware emulation with model-based performance, power, and supply voltage analysis techniques. The DUT, which has to be available in a hardware description language, is integrated into a FPGA along with designated analysis units. These analysis units implement models of the DUT's performance, power consumption, and supply voltage behavior. The presented emulation methodology allows a designer to test designs in such a way that the cycle accurate results are taken online, in real-time, and verify both functional and performance behavior, as well as power consumption and supply voltage levels. The proposed comprehensive emulation methodology is used, as an example of application, to verify the design of a LEON3 multi-core processor system as well as a RF-powered contacatless smart card. The depicted results demonstrate that this emulation approach is suitable to detect functional misbehavior caused by power and supply voltage hazards and how they influence the performance of the system.

Design exploration and evaluation are essential tasks during a product's development cycle. Simulation and hardware emulation are common techniques to explore and evaluate the functionality of hardware/software designs. However, when it comes to distributed secure applications, like contactless reader/smart card systems, non-functional design properties and system aspects (e.g., conctactless power transfer, power consumption) have to be regarded too. State-of-the-art simulation-based and emulation-based design exploration tools cover these design issues and system aspects only to some extent. Here we present a design exploration framework for complete reader/smart card systems using state-of-the-art model-based emulation and estimation techniques. This novel system-based approach is of high importance because of the high availability of battery powered mobile readers (i.e. smart phones) and novel mobile application fields. Contactless power transfer and power consumption analyses of reader and smart cards can be performed for each clock cycle and in real time. Thus, novel system-level power and security optimization techniques can be evaluated considering the reader/smart card system as a whole. We demonstrate the application of our exploration framework by means of a typical Diffie-Hellman key exchange between reader and smart card and highlight power optimization possibilities.

RF-powered smart cards are constrained in their operation by their power consumption. Smart card application designers must pay attention to power consumption peaks, high average power consumption and supply voltage drops. If these hazards are not handled properly, the smart card's operational stability is compromised. Here we present a novel multi-core smart card design, which improves the operational stability of nowadays used smart cards. Estimation based techniques are applied to provide cycle accurate power and supply voltage information of the smart card in real time. A supply voltage management unit monitors the provided power and supply voltage information, flattens the smart card's power consumption and prevents supply voltage drops by means of a dynamic voltage and frequency scaling (DVFS) policy. The presented multi-core smart card design is evaluated on a hardware emulation platform to prove its proper functionality. Experimental tests show that harmful power variations can be reduced by up to 75% and predefined supply voltage levels are maintained properly. The presented analysis and management functionalities are integrated at a minimal area overhead of 10.1%.

Many near field communication (NFC)-based reader / smart card applications are operated at a maximum magnetic field strength to increase the smart card's operational stability. However, a maximum magnetic field strength is worthwhile only in situations of high smart card power requirements (e.g., performing cryptographic operations) or long distance communications. As a result, electrical power is wasted, which limits the run-time of mobile battery-operated reader devices. Here we present an adaptive field strength scaling (AFSS) methodology. The strength of the reader's emitted magnetic field is modified depending on the instantaneous power consumption requirements of the smart card. When the smart card consumes less power, the magnetic field strength is reduced. Whereas when it consumes more power, the magnetic field strength is increased. Thus, the power consumption of the reader / smart card system as a whole is optimized while preserving the smart card's operational stability. In this work, we present the design and implementation of two different AFSS approaches. A reader / smart card hardware emulation platform is used to prove the AFSS technique's feasibility and proper functionality. Experimental tests demonstrate that the energy consumption of the AFSS enhanced reader / smart card system can be reduced by up to 54% compared to current commonly used approaches. Furthermore, we show that the smart card's stability is preserved if the AFSS technique is applied.

Today security and safety applications are often a large conglomerate of complex different components. Because of a strong trend to high system integration to fulfill financial and production cost constraints, as much of these components as possible are combined to form large-scale system-on-chips. Risks of dependability and security problems caused by device degradation and adversaries lead to a wide range of research concerning fault detection and recovery techniques in recent years. Especially in safety systems the concurrent use of different checking techniques, to protect the integrity of the operation, is preferred. Standard duplication or triplication methods for such critical devices are not completely fulfilling this property and raising a need for new on-line testing and recovery methodologies. Furthermore, the smart-card sector produces a strong need for new checking techniques with a low resource footprint. Therefore, this paper presents a novel automatized hardware generation flow to create a predictive control signal monitor unit in an automatized way. Depending on the instruction loaded by the processor pipeline this unit will predict the signature of following control signal changes. Hence, a new way of fault detection, weak checking, is implemented without introducing any large additional hardware blocks. A case study using an open-source processor is also presented to show the applicability of our approach.

Standby power consumption of electric devices is a growing waste of energy. Between 5% and 14% of the residential electrical power consumption is caused by devices being in standby mode. Depending on the device type, more than 50% of standby power consumption could be saved by applying state-of-the-art power management techniques. By implementing a zero energy standby design and outsourcing power consuming user interfaces, even more electrical power can be saved. Here we present a novel Near Field Communication (NFC) interfacing method for everyday electronic devices. By implementing this interface, the target device can be shut down during idle times. Thus, standby power consumption is eliminated completely. If user interaction is requested, NFC provides the electrical energy to switch on the target device's power supply and to start the device. Furthermore, any control, status, and maintenance information can be transmitted over NFC. By outsourcing high power dissipating and unoptimized user interfaces (touch screens, WiFi, etc.) to the power optimized NFC reader, further energy savings are possible also during running state. This paper demonstrates the implementation and integration of this novel interfacing technique into common consumer electronics. Two implementation approaches are presented. A simple, energy harvesting-based approach illustrates the basic working principle. The second, more sophisticated approach, enables also authentication, encrypted data transfer, user interface outsourcing, configuration and control tasks, etc. A proof of concept is demonstrated by means of an access control terminal.