This demonstration presents a high-speed, energy-efficient sensor fusion system integrating CMOS Image Sensors (CIS) and Dynamic Vision Sensors (DVS) for advanced image recognition. Using CIS for high-res imaging and DVS for rapid event-driven capture, the FPGA-implemented architecture with an NPU running YOLOv3-Tiny achieves 18 ms inference latency with a minimal 2.78% mAP loss. Selective NPU activation based on DVS-detected regions yielded 31.5% power savings, while a custom receiver module efficiently fused DVS (13,900 fps) and CIS (60 fps) data. The system uses a power of 6.977 W on a Xilinx Zynq+ ZCU106 board.

Fusing a dynamic vision sensor (DVS) and a CMOS image sensor (CIS) is promising in real-time vision applications. However, unlike common CIS, DVS typically come with a custom data format due to their naturally sparse data, which becomes a challenge to fuse DVS and CIS data streams on a general-purpose CPU. To address this problem, this work proposes a DVS-CIS sensor stream receiver on FPGA. The proposed receiver incorporates a cost-effective address decoder and an inline transpose to receive and store a DVS stream on DRAM effectively. At a system level, a host PC can stream the DVS-CIS stream from FPGA via PCIe and display streams on a monitor. Experimental results demonstrate that our architecture can decode up to 13,900fps of DVS frames without incurring any frame drops while concurrently streaming frames at 60fps from a CIS. The design only uses 135 BRAM, 38 DSPs, 69489 LUTs, and 86626 FFs on a Xilinx Zynq+ ZCU106 FPGA board and consumes a power of 6.977 W.

This study presents a daily surveillance system based on dynamic vision sensors (DVS) and CMOS image sensors (CIS) to enable real-time image recognition with low energy consumption. In such a system, a neural processing unit (NPU) - which executes a DNN model to detect objects on a given CIS image - may consume a lot of energy when always-on. To address the problem, this work introduces a system with a DVS-based region of interest (ROI) detector and an event-based NPU trigger for energy savings. Based on DVS, the ROI detector effectively recognizes scene changes in dynamic environments, e.g., low-light scenes at midnight, which serves as a trigger to invoke the NPU for object detection. Our system prototype was built on a host PC and two Xilinx Zynq+ ZCU106 FPGA boards, one for the DVS-CIS receiver and the other for our NPU. The experimental results demonstrated that Over a 24-hour testing period, our system achieved a 31.5% reduction in energy usage. Operating a YOLOv3-Tiny object detector at 200 MHz, our NPU achieves a latency of just 18 ms, enabling seamless real-time monitoring capabilities.