Event-based cameras are well-suited for fast and agile autonomous navigation due to their ultra-fast, microsecond-level temporal resolution. However, fully leveraging this potential requires highly efficient processing algorithms capable of asynchronous, event-by-event representations and policy updates. Current methods employ synchronous dense representation or process events in a fixed-rate time windows, leading to inefficiencies via redundant computation. We address this by proposing an end-to-end framework for event-to-control policy learning designed for reactive navigation tasks. Our method consists of a memory-augmented perception module that updates the representation asynchronously and adaptively selects the number of events to process. Using the memory representation, a lightweight policy module is jointly optimized with the perception module, and learns to predict control commands at rates that dynamically adjust to scene complexity in an event-based reinforcement learning setting. Evaluations on simulated drone navigation tasks demonstrate higher sample efficiency and robustness compared to dense frame-based methods. Moreover, our approach significantly reduces computational complexity by minimizing processing steps and event counts while maintaining competitive performance against state-of-the-art event-based methods.

In high-speed robotics and autonomous vehicles, rapid environmental adaptation is necessary. Traditional cameras often face issues with motion blur and limited dynamic range. Event-based cameras address these by tracking pixel changes continuously and asynchronously, offering higher temporal resolution with minimal blur. In this work, we highlight our recent efforts in solving the challenge of processing event-camera data efficiently from both algorithm and hardware perspective. Specifically, we present how brain-inspired algorithms such as spiking neural networks (SNNs) can efficiently detect and track object motion from event-camera data. Next, we discuss how we can leverage associative memory structures for efficient event-based represen-tation learning. And finally, we show how our developed Application Specific Integrated Circuit (ASIC) architecture for low-latency, energy-efficient processing outperforms typical GPU/CPU solutions, thus enabling real-time event-based processing. With a 100x reduction in latency and a 1000x lower energy per event compared to state-of-the-art GPU/CPU setups, this enhances the front-end camera systems capability in autonomous vehicles to handle higher rates of event generation, improving control.

In recent years, PIM-based mixed-signal accelerators have been proposed as energy- and area-efficient solutions with ultra-high throughput to accelerate DNN computations. However, PIM designs are sensitive to imperfections such as noise, weight/conductance variations, and cell programming errors that substantially degrade the DNN accuracy. To address this issue, we propose a novel algorithm-hardware co-design framework called Harmonica that simultaneously avoids accuracy degradation due to imperfections, improves area utilization and execution time, and reduces energy consumption. Harmonica proposes to select imperfection-sensitive weights using an input channel-wise method and transfer them to a novel and robust digital accelerator while the main computations are performed in the analog PIM cores. Harmonica is adapted to leverage the preceding weight selection method by reducing ADC precision, employing smaller peripheral circuitry, and a hybrid quantization to optimize the design. Our comprehensive experiments show that even in the presence of imperfections as high as 50%, Harmonica reduces the accuracy degradation from 60% - 90% in designs without a protection solution (e.g., in ISAAC or SRE baselines) to 1% - 2% for different DNNs across diverse datasets. In addition, compared to the ISAAC (SRE), Harmonica improves the execution time, energy, area, power, area-efficiency, and power-efficiency by 26% (14%), 52% (40%), 28% (28%), 57% (45%), 43% (7.5×), and 91% (7.3×), respectively. By employing architecture-based differential cells, where two separated categories of crossbars are used for positive and negative weights, Harmonica outperforms ISAAC (SRE) by 75% (9.2×) and 2.65× (10.2×) in terms of area- and power-efficiency.

Hardware implementations of encryption engines are susceptible to Power Side Channel Attacks (PSCA) and countermeasures only make attacks harder without eliminating them. This work presents a temperature tolerant 65nm CMOS passive on-chip PSCA detection sensor for a AES-128 engine. The design uses on-chip sub-threshold oscillator to detect the resistor normally placed by an attacker on the power-line of an AES chip to mount PSCA. The measurement demonstrates 99.9% probability of successful detection, 1.1ms detection time, minimal area overhead compared to the encryption area and 0.83mW power.

Neuromorphic event-based cameras can unlock the true potential of bio-plausible sensing systems that mimic our human perception. However, efficient spatiotemporal processing algorithms must enable their low-power, low-latency, real-world application. In this talk, we highlight our recent efforts in this direction. Specifically, we talk about how brain-inspired algorithms such as spiking neural networks (SNNs) can approximate spatiotemporal sequences efficiently without requiring complex recurrent structures. Next, we discuss their event-driven formulation for training and inference that can achieve realtime throughput on existing commercial hardware. We also show how a brain-inspired recurrent SNN can be modeled to perform on event-camera data. Finally, we will talk about the potential application of associative memory structures to efficiently build representation for event-based perception.

Derailments present a frequent phenomenon in several developing countries, which result in massive loss of property along with death tolls. For preventing derailments, a real-time automated system is needed to detect uprooted or faulty rail blocks. One of the solutions in this context is to sense the vibration of the rail track having an incoming train and transmit the information to the train notifying it about the condition of the rail track ahead. However, existing studies in this regard are yet to present a pragmatic solution that enables much-demanded long-distance networking to transmit the sensed data. The demand for long-distance network communication between the sensor nodes and the incoming train is unavoidable, as stopping the train after sensing an uprooted or faulty rail block ahead needs a considerable response time and distance. Therefore, in this paper, we develop a low-cost, long-range, and highly reliable mobile multi-hop networking scheme to successfully transmit data sensed from rail tracks to an approaching train at a distance of around 2000m. By considering the effect of Fresnel’s Region in our study, we determine the suitable placement of the networking module on the rail track, which leads us to achieve a delivery ratio of more than 99%. We confirm this finding through rigorous experiments over a real testbed scenario enabling mobile multi-hop networking.

Thoracic disease detection from chest radiographs using deep learning methods has been an active area of research in the last decade. Most previous methods attempt to focus on the diseased organs of the image by identifying spatial regions responsible for significant contributions to the model's prediction. In contrast, expert radiologists first locate the prominent anatomical structures before determining if those regions are anomalous. Therefore, integrating anatomical knowledge within deep learning models could bring substantial improvement in automatic disease classification. Motivated by this, we propose Anatomy-XNet, an anatomy-aware attention-based thoracic disease classification network that prioritizes the spatial features guided by the pre-identified anatomy regions. We adopt a semi-supervised learning method by utilizing available small-scale organ-level annotations to locate the anatomy regions in large-scale datasets where the organ-level annotations are absent. The proposed Anatomy-XNet uses the pre-trained DenseNet-121 as the backbone network with two corresponding structured modules, the Anatomy Aware Attention (A3) and Probabilistic Weighted Average Pooling, in a cohesive framework for anatomical attention learning. We experimentally show that our proposed method sets a new state-of-the-art benchmark by achieving an AUC score of 85.78%, 92.07%, and, 84.04% on three publicly available large-scale CXR datasets–NIH, Stanford CheXpert, and MIMIC-CXR, respectively. This not only proves the efficacy of utilizing the anatomy segmentation knowledge to improve the thoracic disease classification but also demonstrates the generalizability of the proposed framework.

Robust traffic sign detection and recognition (TSDR) is of paramount importance for the successful realization of autonomous vehicle technology. The importance of this task has led to vast amount of research efforts and many promising methods have been proposed in the existing literature. However, most of these methods have been evaluated on clean and challenge-free datasets and overlooked the performance deterioration associated with different challenging conditions (CCs) that obscure the traffic-sign images captured in the wild. In this paper, we look at the TSDR problem under CCs and focus on the performance degradation associated with them. To this end, we propose a Convolutional Neural Network (CNN) based prior enhancement focused TSDR framework. Our modular approach consists of a CNN-based challenge classifier, Enhance-Net–an encoder-decoder CNN architecture for image enhancement, and two separate CNN architectures for sign-detection and classification. We propose a novel training pipeline for Enhance-Net that focuses on the enhancement of the traffic sign regions (instead of the whole image) in the challenging images subject to their accurate detection. We used CURE-TSD dataset consisting of traffic videos captured under different CCs to evaluate the efficacy of our approach. We experimentally show that our method obtains an overall precision and recall of 91.1% and 70.71% that is 7.58% and 35.90% improvement in precision and recall, respectively, compared to the current benchmark. Furthermore, we compare our approach with different CNN-based TSDR methods and show that our approach outperforms them by a large margin.

Traffic sign detection is a central part of autonomous vehicle technology. Recent advances in deep learning algorithms have motivated researchers to use neural networks to perform this task. In this paper, we look at traffic sign detection as an image segmentation problem and propose a deep convolutional neural network-based approach to solve it. To this end, we propose a new network, the SegU-Net, which we form by merging the state-of-the-art segmentation architectures-SegNet and U-Net to detect traffic signs from video sequences. For training the network, we use the Tversky loss function constrained by an L1 term instead of the intersection over union loss traditionally used to train segmentation networks. We use a separate network, inspired by the VGG-16 architecture, to classify the detected signs. The networks are trained on the challenge free sequences of the CURE-TSD dataset. Our proposed network outperforms the state-of-the-art object detection networks, such as the Faster R-CNN inception Resnet V2 and R-FCN Resnet 101, by a large margin and obtains a precision and recall of 94.60% and 80.21%, respectively, which is the current state of the art on this part of the dataset. In addition, the network is tested on the German Traffic Sign Detection Benchmark (GTSDB) dataset, where it achieves a precision and recall of 95.29% and 89.01%, respectively. This is on a par with the performance of the aforementioned object detection networks. These results prove the generalizability of the proposed architecture and its suitability for robust traffic sign detection in autonomous vehicles.

Detecting count of human beings accurately in a closed indoor environment is crucial in diverse application areas including search and rescue, surveillance, customer analytics, abnormal event detection, human gait characterization, congestion analysis and many more. Moreover, it has significant importance in preventing any intrusion in a secured indoor space such as a bank vault. Sensors-based technologies (for example camera, PR, etc.) are becoming more popular day by day as the regular methodologies are not good enough to ensure enhanced security in a closed indoor environment. As sensors used in these technologies have to be deployed in visible places, there exist possibilities of damaging the sensors by the intruder. Therefore, this paper proposes a novel methodology to detect human count in such closed indoor setting, which can be deployed in any hidden place. Here, human count is done based on four environmental gaseous parameters (Carbon Dioxide, Liquefied Petroleum Gas or LPG, Nitrogen Dioxide, and Sulfur Dioxide) and two weather parameters (temperature and humidity). Real experiments are done under closed controlled settings and counting is done using machine learning algorithms such as Bagging, Random-Forest, IBK, and J48. We achieve more than 99% accuracy for some of the classifiers in detecting the number of humans present.