Recent studies have found that the position of mice or rats can be decoded from calcium imaging of brain activity offline. However, given the complex analysis pipeline, real-time position decoding remains a challenging task, especially considering strict requirements on hardware usage and energy cost for closed-loop feedback applications. In this paper, we propose two neural network based methods and corresponding hardware designs for real-time position decoding from calcium images. Our methods are based on: 1) convolutional neural network (CNN), 2) spiking neural network (SNN) converted from the CNN. We implemented quantized CNN and SNN models on FPGA. Evaluation results show that the CNN and the SNN methods achieve 56.3%/83.1% and 56.0%/82.8% Hit-1/Hit-3 accuracy for the position decoding across different rats, respectively. We also observed an accuracy-latency tradeoff of the SNN method in decoding positions under various time steps. Finally, we present our SNN implementation on the neuromorphic chip Loihi.

Miniaturized fluorescent calcium imaging miniscope has become a prominent technique in monitoring the activity of a large population of neurons in vivo. However, existing calcium image processing algorithms are developed for off-line analysis, and their implementations on general-purpose processors are difficult to meet the real-time processing requirement under constrained energy budget for closed-loop applications. In this paper, we propose the CANSEE, a customized accelerator for neural signal enhancement and extraction from calcium image in real time. The accelerator can perform the motion correction, the calcium image enhancement, and the fluorescence tracing from up to 512 cells with less than 1-ms processing latency. We also designed the hardware that can detect new cells based on the long short-term memory (LSTM) inference. We implemented the accelerator on a Xilinx Ultra96 FPGA. The implementation achieves 15.8x speedup and over 2 orders of magnitude improvement in energy efficiency compared to the evaluation on the multi-core CPU.

Miniaturized fluorescent calcium imaging microscopes are widely used for monitoring the activity of a large population of neurons in freely behaving animals in vivo. Conventional calcium image analyses extract calcium traces by iterative and bulk image processing and they are hard to meet the power and latency requirements for neurofeedback devices. In this paper, we propose the calcium image processing pipeline based on a bit-sparse long short-term memory (LSTM) inference kernel (BLINK) for efficient calcium trace extraction. It largely reduces the power and latency while remaining the trace extraction accuracy. We implemented the customized pipeline on the Ultra96 platform. It can extract calcium traces from up to 1024 cells with sub-ms latency on a single FPGA device. We designed the BLINK circuits in a 28-nm technology. Evaluation shows that the proposed bit-sparse representation can reduce the circuit area by 38.7% and save the power consumption by 38.4% without accuracy loss. The BLINK circuits achieve 410 pJ/inference, which has 6293x and 52.4x gains in energy efficiency compared to the evaluation on the high performance CPU and GPU, respectively.