Omnidirectional videos (ODVs) provide an immersive visual experience by capturing the 360° scene. With the rapid advancements in virtual/augmented reality, metaverse, and generative artificial intelligence, the demand for high-quality ODVs is surging. However, ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth. Although video super-resolution (SR) is a capable video quality enhancement technique, the performance ceiling and practical generalization of existing methods are limited when applied to ODVs due to their unique attributes. To alleviate spatial projection distortions and temporal flickering of ODVs, we propose a Spatio-Temporal Distortion Aware Network (STDAN) with joint spatio-temporal alignment and reconstruction. Specifically, we incorporate a spatio-temporal continuous alignment (STCA) to mitigate discrete geometric artifacts in parallel with temporal alignment. Subsequently, we introduce an interlaced multi-frame reconstruction (IMFR) to enhance temporal consistency. Furthermore, we employ latitude-saliency adaptive (LSA) weights to focus on regions with higher texture complexity and human-watching interest. By exploring a spatio-temporal jointly framework and real-world viewing strategies, STDAN effectively reinforces spatio-temporal coherence on a novel ODV-SR dataset and ensures affordable computational costs. Extensive experimental results demonstrate that STDAN outperforms state-of-the-art methods in improving visual fidelity and dynamic smoothness of ODVs.

Deformable image registration (DIR) is an active research topic in biomedical imaging. There is a growing interest in developing DIR methods based on deep learning (DL). A traditional DL approach to DIR is based on training a convolutional neural network (CNN) to estimate the registration field between two input images. While conceptually simple, this approach comes with a limitation that it exclusively relies on a pre-trained CNN without explicitly enforcing fidelity between the registered image and the reference. We present plug-and-play image registration network (PIRATE) as a new DIR method that addresses this issue by integrating an explicit data-fidelity penalty and a CNN prior. PIRATE pre-trains a CNN denoiser on the registration field and "plugs" it into an iterative method as a regularizer. We additionally present PIRATE+ that fine-tunes the CNN prior in PIRATE using deep equilibrium models (DEQ). PIRATE+ interprets the fixed-point iteration of PIRATE as a network with effectively infinite layers and then trains the resulting network end-to-end, enabling it to learn more task-specific information and boosting its performance. Our numerical results on OASIS and CANDI datasets show that our methods achieve state-of-the-art performance on DIR.

Benefiting from the 360 • field of view (FoV) of the omnidirectional images (ODIs), users could enjoy an immersive experience with head-mounted devices or computers.High-resolution ODIs can provide pleasing visual experience and boost the performance of related visual tasks.Therefore, Super-resolution (SR) is an essential technique during the application of ODIs.However, traditional SR methods fail to enhance the most widely utilized equirectangular projection (ERP) format ODIs due to projection distortions.Existing ODI-SR methods take the latitude-related position information as a prior, but lack the adaptation to the ERP content distribution characteristics.To address this issue, we propose a novel Frequency Attention Transformer ODI-SR (FATO) network focusing on highfrequency details of ODIs.In particular, we transform an ODI into fine-grained patches in the frequency domain through Discrete Cosine Transform (DCT).After that, we design a frequency selfattention mechanism to capture the relationship between different frequency patches.Subsequently, we introduce a frequency loss function to further constrain the network.Extensive experimental results demonstrate that the proposed FATO achieves superior performance over state-of-the-art methods on ODIs.

Spiking Neural Networks (SNNs) are energy-efficient artificial neural network models that can carry out data-intensive applications. Energy consumption, latency, and memory bottleneck are some of the major issues that arise in machine learning applications due to their data-demanding nature. Memristor-enabled Computing-In-Memory (CIM) architectures have been able to tackle the memory wall issue, eliminating the energy and time-consuming movement of data. In this work we develop a scalable CIM-based SNN architecture with our fabricated two-layer memristor crossbar array. In addition to having an enhanced heat dissipation capability, our memristor exhibits substantial enhancement of 10% to 66% in design area, power and latency compared to state-of-the-art memristors. This design incorporates an inter-spike interval (ISI) encoding scheme due to its high information density to convert the incoming input signals into spikes. Furthermore, we include a time-to-first-spike (TTFS) based output processing stage for its energy-efficiency to carry out the final classification. With the combination of ISI, CIM and TTFS, this network has a competitive inference speed of 2μs/image and can successfully classify handwritten digits with 2.9mW of power and 2.51pJ energy per spike. The proposed architecture with the ISI encoding scheme can achieve ∼10% higher accuracy than those of other encoding schemes in the MNIST dataset.

Omnidirectional image (ODI) super-resolution (SR) is an important technique in augmented reality and virtual reality applications to address the low-resolution problem caused by limitations in capturing devices or bandwidth. The ODI projection distortion makes it challenging to apply existing SR methods. In this paper, we propose an ODI SR method by leveraging the characteristics of ODIs and human visual characteristics. Specifically, we firstly design a perception-orientated adaptive loss function by jointly utilizing saliency map and latitude map. In our proposed ODI-SR network, we introduce an attention module to aggregate multi-scale information and leverage spherical convolution to adapt to the spheric format of ODIs. Furthermore, we design a data augmentation strategy for ODIs according to viewpoint distribution to further improve the visual quality of SR images. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance according to both qualitative and quantitative evaluations.

Plug-and-play (PnP) prior is a well-known class of methods for solving imaging inverse problems by computing fixed-points of operators combining physical measurement models and learned image denoisers. While PnP methods have been extensively used for image recovery with known measurement operators, there is little work on PnP for solving blind inverse problems. We address this gap by presenting a new block-coordinate PnP (BC-PnP) method that efficiently solves this joint estimation problem by introducing learned denoisers as priors on both the unknown image and the unknown measurement operator. We present a new convergence theory for BC-PnP compatible with blind inverse problems by considering nonconvex data-fidelity terms and expansive denoisers. Our theory analyzes the convergence of BC-PnP to a stationary point of an implicit function associated with an approximate minimum mean-squared error (MMSE) denoiser. We numerically validate our method on two blind inverse problems: automatic coil sensitivity estimation in magnetic resonance imaging (MRI) and blind image deblurring. Our results show that BC-PnP provides an efficient and principled framework for using denoisers as PnP priors for jointly estimating measurement operators and images.

Three-dimensional integrated circuits (3D-ICs) is a cutting-edge design methodology of placing the circuitry vertically aiming for a high-speed and energy-efficient system with the smallest design area. In this article, a novel 3-D neuromorphic system is proposed and analyzed, which utilizes the fabricated two-layer memristor as the electronic synapses in a spiking neural network (SNN). The two-layer structure of the memristors leads to a significant improvement in the design area ($2\times $), power consumption ($1.48 \times $), and latency ($2.58 \times $), compared to the traditional one-layer configuration. Meanwhile, the heat dissipation layers are added to our memristors reducing 30% cycle-to-cycle switching variation. Our memristive synapses are utilized for storing the exported weights of the SNNs that have threshold function as the activation function. The proposed neuromorphic system is evaluated using a hardware–software co-design approach importing the weights of SNNs into NeuroSIM. The simulation results demonstrate the significant improvement of memristive synapses on design area, power consumption, and latency, compared with the static random-access memory (SRAM) and other state-of-the-art memristive synapses (10%–66%).

Deep neural networks for medical image reconstruction are traditionally trained using high-quality ground-truth images as training targets. Recent work on Noise2Noise (N2N) has shown the potential of using multiple noisy measurements of the same object as an alternative to having a ground-truth. However, existing N2N-based methods are not suitable for learning from the measurements of an object undergoing nonrigid deformation. This paper addresses this issue by proposing the deformation-compensated learning (DeCoLearn) method for training deep reconstruction networks by compensating for object deformations. A key component of DeCoLearn is a deep registration module, which is jointly trained with the deep reconstruction network without any ground-truth supervision. We validate DeCoLearn on both simulated and experimentally collected magnetic resonance imaging (MRI) data and show that it significantly improves imaging quality.

Deep neural networks (DNNs), a brain-inspired learning methodology, requires tremendous data for training before performing inference tasks. The recent studies demonstrate a strong positive correlation between the inference accuracy and the size of the DNNs and datasets, which leads to an inevitable demand for large DNNs. However, conventional memory techniques are not adequate to deal with the drastic growth of dataset and neural network size. Recently, a resistive memristor has been widely considered as the next generation memory device owing to its high density and low power consumption. Nevertheless, its high switching resistance variations (cycle-to-cycle) restrict its feasibility in deep learning. In this work, a novel memristor configuration with the enhanced heat dissipation feature is fabricated and evaluated to address this challenge. Our experimental results demonstrate our memristor reduces the resistance variation by ~ 30% and the inference accuracy increases correspondingly in a similar range. The accuracy increment is evaluated by our deep delay-feed-back reservoir computing (Deep-DFR) model. The design area, power consumption, and latency are reduced by ~48%, ~42%, and ~67%, respectively, compared to the conventional static random-access memory technique (6T). The performance of our memristor is improved at various degrees (~13%-73%) compared to the state-of-the-art memristors.

An advanced neurophysiological computing system can incorporate a 3-D integration system composed of emerging nano-scale devices to provide massive parallelism having high speed, low cost, and energy efficient hardware implementation. Due to process technology constraints, a certain amount of redundant through silicon vias (TSVs) and dummy TSVs are always required in a 3-D integrated system. In this paper, we propose to use these redundant and dummy TSVs to supply the neuronal membrane capacitance that maps the membrane electrical activity in a hybrid 3-D neuromorphic system. This proposition could also serve the need of neuronal ion transportation dynamics. We also investigate two new methodologies that could significantly enhance the TSV capacitance in a 3-D neuromorphic system. The capacitance of these enhanced TSVs is studied with analytical models; the accuracy of the models are evaluated against 3-D field extracted values. The advantage of using the TSVs to mimic membrane capacitance in a 3-D neuromorphic chip is demonstrated through comparisons of both silicon area and energy consumption against their 2-D counterpart designs.

Neurophysiological architecture using 3D integration technology offers a high device interconnection density as well as fast and energy efficient links among the neuron and synapses layers. In this paper, we propose to reconfigure the Through-Silicon-Vias (TSVs) to serve as the neuronal membrane capacitors that map the membrane electrical activities in a hybrid 3D neuromorphic system. We also investigate new methodology that could significantly enhance the TSV capacitance to achieve a high efficiency of signal processing through membrane. An optimal CAD framework is designed to optimally utilize such TSV devices, and resolve the signal-integrity issues arising at fast data rates during massively parallel data transmissions. The electrical performance of the 3D neuromorphic chip is compared against the ones of the 2D counterpart design to demonstrate the advantages of our design and methodology.

In the recent MRI scanning, ultra-high-field (7T) MR imaging provides higher resolution and better tissue contrast compared to routine 3T MRI, which may help in more accurate and early brain diseases diagnosis. However, currently, 7T MRI scanners are more expensive and less available at clinical and research centers. These motivate us to propose a method for the reconstruction of images close to the quality of 7T MRI, called 7T-like images, from 3T MRI, to improve the quality in terms of resolution and contrast. By doing so, the post-processing tasks, such as tissue segmentation, can be done more accurately and brain tissues details can be seen with higher resolution and contrast. To do this, we have acquired a unique dataset which includes paired 3T and 7T images scanned from same subjects, and then propose a hierarchical reconstruction based on group sparsity in a novel multi-level Canonical Correlation Analysis (CCA) space, to improve the quality of 3T MR image to be 7T-like MRI. First, overlapping patches are extracted from the input 3T MR image. Then, by extracting the most similar patches from all the aligned 3T and 7T images in the training set, the paired 3T and 7T dictionaries are constructed for each patch. It is worth noting that, for the training, we use pairs of 3T and 7T MR images from each training subject. Then, we propose multi-level CCA to map the paired 3T and 7T patch sets to a common space to increase their correlations. In such space, each input 3T MRI patch is sparsely represented by the 3T dictionary and then the obtained sparse coefficients are used together with the corresponding 7T dictionary to reconstruct the 7T-like patch. Also, to have the structural consistency between adjacent patches, the group sparsity is employed. This reconstruction is performed with changing patch sizes in a hierarchical framework. Experiments have been done using 13 subjects with both 3T and 7T MR images. The results show that our method outperforms previous methods and is able to recover better structural details. Also, to place our proposed method in a medical application context, we evaluated the influence of post-processing methods such as brain tissue segmentation on the reconstructed 7T-like MR images. Results show that our 7T-like images lead to higher accuracy in segmentation of white matter (WM), gray matter (GM), cerebrospinal fluid (CSF), and skull, compared to segmentation of 3T MR images.

In this paper, we propose a new bootstrap scheme, called the nonlocal bootstrap (NLB) for uncertainty estimation. In contrast to the residual bootstrap, which relies on a data model, or the repetition bootstrap, which requires repeated signal measurements, NLB is not restricted by the data structure imposed by a data model and obviates the need for time-consuming multiple acquisitions. NLB hinges on the observation that local imaging information recurs in an image. This self-similarity implies that imaging information coming from spatially distant (nonlocal) regions can be exploited for more effective estimation of statistics of interest. Evaluations using in silico data indicate that NLB produces distribution estimates that are in closer agreement with those generated using Monte Carlo simulations, compared with the conventional residual bootstrap. Evaluations using in vivo data demonstrate that NLB produces results that are in agreement with our knowledge on white matter architecture.

In dynamic susceptibility contrast perfusion weighted imaging, the recirculation effect is normally removed by gamma-variate fitting from concentration curves before estimating hemodynamic parameters. At lower SNR, however, many fitting failures may result. Moreover, when cerebral hemodynamics is compromised e.g., cerebral ischemia, a substantially broadened concentration curve is anticipated, resulting in the first passage overlapping with recirculation, which again causes a gamma-fit to fail to consistently discern recirculation contributions from the first passage. We propose to exploit independent component analysis to obviate the recirculation effect. We demonstrate that such a technique can remove recirculation in normal and ischemic brain tissues while preserving the first passage. This in turn allows for accurate recirculation elimination and hence improved estimation of cerebral blood volume particularly when overlapping between first passage and recirculation is suspected as in the case of an ischemic lesion.