Heterogeneous sensors (e.g., 2D cameras and LiDAR) provide a novel opportunity to leverage multiple modalities in collaborative artificial intelligence (AI)-based video analytics pipelines. Such applications use sensors that are frequently attached to resource-limited edge devices which can hinder the execution of multimodal and deep DNN models. While powerful edge devices can still benefit from multimodal fusion to enhance robustness, joint training of such models for generalizable applications is often infeasible due to the lack of large-scale multimodal datasets and the prohibitive cost involved in annotating those datasets. To address this, we introduce FusionBridge: a lightweight fusion framework that combines the capabilities of independently trained 2D (image-based) and 3D (LiDAR-based) perception models to improve object detection at the edge. FusionBridge extracts mid-level features from single modality 3D models and performs cross-modal fusion via a lightweight transformer-based adapter. This enables hints to be exchanged without requiring joint end-to-end training. By bridging modality-specific experts, our approach maintains modularity, supports model reuse, and allows scalable deployment across heterogeneous sensor configurations with zero calibration or sensor alignment effort. Evaluations on simulated and real world deployments demonstrate that FusionBridge achieves up to a 57% F1-score improvement over any single-modality baseline, while only incurring a 15% latency overhead and 0.4KB/frame transmission overhead compared to the baseline.

While DNN models have dramatically improved the accuracy of machine vision tasks, pervasive deployments of vision sensors for surveillance tasks continue to suffer from high energy consumption and network traffic overhead. To tackle these problems, we introduce CollabCam , an edge-based machine vision system designed for multi-camera deployments that leverages the naturally-occurring overlaps in the field-of-view (FoV) among neighboring cameras. CollabCam synergistically combines two innovative ideas: (a) having each individual camera compose and transmit mixed-resolution frames (MRF) via lightweight down-sampling, where the transmitted images have significantly lower resolution in the shared, overlapping portions , and (b) performing inference, for an exemplar object detection task, for each camera stream using a new collaborative mechanism which utilizes suitably-translated object bounding boxes from a peer “collaborating” camera as an additional input channel. We demonstrate how this collaborative mechanism is generalizable and can be realized by simply retraining off-the-shelf object detector DNNs, such as YOLOv3 and SSD, without modifying their model structures. By emulating the performance of CollabCam using two benchmark outdoor-campus multi-camera datasets, we show that Collab-DNNs can accommodate a 50–60 fold reduction in image size (therefore reducing network transmission overhead), for both high-resolution (1056x1056) and low-resolution (512x512) images, with a modest ≤ 2 - 5% drop in object detection accuracy, compared to a non-collaborative approach that suffers a ∼ 45–60% drop in accuracy. Subsequently, by deploying a Raspberry-Pi based CollabCam prototype on a campus-based test-bed, we demonstrate that CollabCam can reduce the overall energy/image frame overhead by ∼25–35%, with even higher energy savings (∼35–45%) likely with hardware optimization. Finally, additional experiments help demonstrate that CollabCam can prove beneficial for varied (including multi-class) object detection tasks and that CollabCam’s performance benefits may be best realized by ensuring that the number of deployed, collaborating cameras is not excessively high.

While Deep Neural Network (DNN) models have transformed machine vision capabilities, their extremely high computational complexity and model sizes present a formidable deployment roadblock for AIoT applications. We show that the complexity-vs-accuracy-vs-communication tradeoffs for such DNN models can be significantly addressed via a novel, lightweight form of "collaborative machine intelligence" that requires only runtime changes to the inference process. In our proposed approach, called ComAI, the DNN pipelines of different vision sensors share intermediate processing state with one another, effectively providing hints about objects located within their mutually-overlapping Field-of-Views (FoVs). CoMAI uses two novel techniques: (a) a secondary shallow ML model that uses features from early layers of a peer DNN to predict object confidence values in the image, and (b) a pipelined sharing of such confidence values, by collaborators, that is then used to bias a reference DNN’s outputs. We demonstrate that CoMAI (a) can boost accuracy (recall) of DNN inference by 20-50%, (b) works across heterogeneous DNN models and deployments, and (c) incurs negligible processing, bandwidth and processing overheads compared to non-collaborative baselines.