CNN-based stereo matching methods achieve great performance but come with high computational requirements. Pruning a CNN can reduce the complexity but may in turn lead to memory conflicts, lowering throughput. Our proposed architecture and memory mapping technique aim at reducing conflicts to exploit extremely sparse stereo matching networks. To maintain a high utilization of processing elements, we decompose the de-convolution operation into several convolution operations. The proposed architecture provides a 2.1 x speed up over SCNN. Compared to the software implementation, only 0.01% performance drop is observed, so that the proposed architecture obtains state-of-the-art accuracy compared to existing sparsity aware hardware implementations.

Sending local data to cloud servers is vulnerable to user privacy, and its long update latency. Meanwhile, the state-of- the-art stereo matching method is still computation demanding, fine-tuning the whole model on-device is not a practicable solution because of the limited power budget and computation ability on edge devices. In this study, we propose a two-stage online stereo matching refinement system, using an additional light-weight network to learn the domain gap between local data and cloud training data. We define a load-gain ratio to evaluate computer efficiency. This refinement system has a much better load-gain ratio than fine-tune. (0.2 v.s. 35.7 operation overhead/accuracy gain) Nevertheless, we only disburse 0.2% of additional parameters and 0.7% additional computation as set by inference the stereo matching model. Thus, it would be a suitable choice for an online training scenario. With re-scheduling the training pipeline, we use a patch-based layer fusion technique and reduce the off-chip memory bandwidth by 97%.

In this study, we propose a constant memory hardware architecture that can support weighted mode, median, and joint bilateral filters, which is referred to as CMWMF. This work aims to meet the high memory and computation requirements of processing depth maps with a large number of depth candidates. In the proposed architecture, we leverage the geometry smoothing characteristic of natural images to reduce the static random access memory (SRAM) size for hardware implementation. The architecture preserves a constant number of disparity values instead of depending on the label count and size of the local supporting window. A novel weighted median search procedure is proposed, which assigns a computation to each input cycle, thereby rendering the process hardware friendly. An index-checking technique is proposed to process out-of-order joint histograms. We adopted the above-mentioned techniques in our architecture as they consume a constant SRAM size and supports multiple types of filters. As a result, this architecture reduces the SRAM size by 92.4% with a negligible decrease in performance. According to our analysis on the KITTI, and Middlebury datasets, and with actual depth cameras, the preserved information is sufficient. The proposed architecture is one of the most suitable depth refinement architectures for scenarios having a large number of depth candidates.

Belief propagation (BP)-based stereo matching has popular owing to its regularity and ability to yield promising results. Some commonly observed hardware-implementation challenges pertaining to the use of this algorithm are large memory requirements and trade-offs between speed and chip area, along with an increasing disparity range. The paper presents a hardware- and memory-efficient architecture for building a BP-based disparity estimation system capable of overcoming issues associated with large disparity ranges. The proposed architecture is memory-efficient owing to the regularity of its underlying algorithm. In addition, the improved hardware efficiency can be attributed to processing element modifications to demonstrate shareable characteristics. Results obtained in this study reveal a 67.8% reduction in required memory corresponding to a time-area term complexity of O(L(logL)2), where L denotes the disparity range. This result is in stark contrast to the O(L2logL) and O(L2) complexities observed in extant studies. Compared to state-of-the-art implementations, the proposed architecture offers an 86.2% gate count reduction for message update units at a disparity range of 512. These results confirm the proposed architecture's suitability for use in large disparity scenarios.

The belief propagation (BP) technique is successful in image stereo matching problem. However, when we consider stereo matching for videos, directly applying the BP algorithm frame by frame results in unsatisfactory temporally inconsistent disparity maps. In this paper, we present the temporally consistent belief propagation for video stereo matching. We introduce a temporal term in traditional BP objective function and propose an adaptive weighting scheme to account for this temporal term. We show that the proposed algorithm performs favorably against previous methods in the stereo video datasets. Furthermore, the proposed method can solve problems induced by previous methods like error propagation from previously occluded regions.

CNN-based super-resolution methods achieve great performance but come with high bandwidth requirements. This paper proposes a bandwidth efficient CNN-based architecture for super resolution supporting up to Full HD images at 60 fps. The bandwidth efficiency is achieved by layer fusion with data reuse scheduling and dynamic quantization. To maintain a high utilization of processing elements, the deconvolution operation is decomposed into a several convolution operations. The proposed architecture provides a 94.8 % bandwidth reduction. Compared to the software implementation, less than 1 % performance drop is induced in this VLSI implementation. The enables the proposed architecture to obtain state-of-the-art accuracy compared to existing super-resolution hardware implementations.

An accurate segment-based cost aggregation method is proposed in this paper. Segment-based methods can speed up the procedure according to the assumption that pixels belonging the same segment should share the same physical property. One of the most critical problems of those methods is the limited pixels within the corresponding segment. To enlarge supporting region of aggregation efficiently, we include similar adjacent segments with a fixed weight instead of a complex one. With introducing the simplified spatial weight for adjacent segments, the proposed method outperforms existing segment-based cost aggregate method[11], the Adaptive Support-weight[3], and the cross-based cost aggregation[10]. Comparing the other aspect of performance, this algorithm also speeds up the computation time by 43%.

This paper introduces a memory efficient architecture for belief propagation based disparity estimation. To find the bottleneck of the memory, a lifetime analysis of the exchanged message is presented. The analysis leads to architecture which can take advantage of the data characteristics resulting in memory reduction, computing resource and memory access. In the experimental result, the proposed architecture gains 20% speed up in software simulation compared to non-optimized counterpart. In hardware implementation, more than 42% area is reduced by this architecture design without affecting the performance. With further design effort 61.8% area reduction is achieved.

One memory-logic-integration design platform is developed in this paper with thermal reliability analysis provided for 2.5D through-silicon-interposer (TSI) and 3D through-silicon-via (TSV) based integrations. Temperature-dependent delay and power models have been developed at microarchitecture level for 2.5D and 3D integrations of many-core microprocessors and main memory, respectively. Experiments are performed by general-purpose benchmarks from SPEC CPU2006 and also cloud-oriented benchmarks from Phoenix with the following observations. The memory-logic integration by 3D RC-interconnected TSV I/Os can result in thermal runaway failures due to strong electrical-thermal couplings. On the other hand, the one by 2.5D transmission-line-interconnected TSI I/Os has shown almost the same energy efficiency and better thermal resilience.

Designing of arbiters has become increasingly important due to their wide use in the areas such as multi-processor systems-on-a-chip and on-chip or off-chip high-speed cross-bar switches and networks. In this paper, we proposed a new systematic model-driven flow for designing the new scalable multi-facet arbiters through a 3-phase process combined with the template-based modular design approach that includes the design model derivation phase, the architecture model (or template) design phase as well as the arbiter hardware implementation and generation phase. First, we described the phase 1 of the design flow of how to induce an arbiter design model in detail by careful analysis of arbiter design issues and systematic design space exploring the construction of the model. Then, we continue to discuss the phase 2 of how to derive an architecture model or template using the reusability, modularity and expansibility techniques. With both the design and the architecture models, designers can easily design or at least understand and thus choose many kinds of different but better arbiters efficiently. Finally, we have developed a scalable decentralized parallel tree structure and corresponding modular algorithms for efficient arbiter hardware implementation, which also is the final phase of our proposed 3-phase model-driven design flow. Moreover, based on this modular and reusable hardware implementation structure as well as the algorithms, we have designed a parametric arbiter generator that automatically generates various multi-facet arbiters. Using this hardware implementation design and the generator, not only a fast and small round-robin arbiter but also other type arbiters were designed and generated on the fly. The hardware implementation algorithms, the generator, and the experiment results all were given to verify their performances. To our knowledge, this is the first time that such a systematic model-driven design approach is proposed in the practical hardware design and such a multi-facet arbiter is designed.

Designing of arbiters has become increasingly important because of their wide use in such as multi-processor system-on-a-chips (MPSoCs) and on- or off- chip high-speed cross-bar switches and networks. In this and an accompanying paper [6], we proposed a new systematic model-driven flow for design of the proposed new scalable multi-facet arbiters through a 3-phase process combined with the template-based modular design approach: the design model derivation phase, the architecture model (or template) design phase as well as arbiter hardware implementation and generation phase. Here, we described the phase 1 of the design flow of how to induce an arbiter design model in detail by careful analysis of arbiter design issues and systematic design space exploring the construction of the model. Then, we continue to discuss the phase 2 of how to derive an architecture model or template using the reusability, modularity and expansibility techniques. With both the design and architecture models, designers can easily design or at least understand and thus choose many kinds of different but better arbiters efficiently. The phase 3 is described in the accompanying paper [6]. To our knowledge, this is the first time that such a systematic model-driven design approach is proposed for the practical hardware design.