Scene graph generation (SGG) aims to understand sophisticated visual information by detecting triplets of subject, object, and their relationship (predicate). Since the predicate labels are heavily imbalanced, existing supervised methods struggle to improve accuracy for the rare predicates due to insufficient labeled data. In this paper, we propose SePiR, a novel self-supervised learning method for SGG to improve the representation of rare predicates. We first train a relational encoder by contrastive learning without using predicate labels, and then fine-tune a predicate classifier with labeled data. To apply contrastive learning to SGG, we newly propose data augmentation in which subject-object pairs are augmented by replacing their visual features with those from other images having the same object labels. By such augmentation, we can increase the variation of the visual features while keeping the relationship between the objects. Comprehensive experimental results on the Visual Genome dataset show that the SGG performance of SePiR is comparable to the state-of-theart, and especially with the limited labeled dataset, our method significantly outperforms the existing supervised methods. Moreover, SePiR’s improved representation enables the model architecture simpler, resulting in 3.6x and 6.3x reduction of the parameters and inference time from the existing method, independently.

In this work, we propose ENSEI, a secure inference (SI) framework based on the frequency-domain secure convolution (FDSC) protocol for the efficient execution of image inference in the encrypted domain. Our observation is that, under the combination of homomorphic encryption and secret sharing, homomorphic convolution can be obliviously carried out in the frequency domain, significantly simplifying the related computations. We provide protocol designs and parameter derivations for number-theoretic transform (NTT) based FDSC. In the experiment, we thoroughly study the accuracy-efficiency trade-offs between time- and frequency-domain homomorphic convolution. With ENSEI, compared to the best known works, we achieve 5--11x online time reduction, up to 33x setup time reduction, and up to 10x reduction in the overall inference time. A further 33% of bandwidth reductions can be obtained on binary neural networks with only 3% of accuracy degradation on the CIFAR-10 dataset.

A secure version of the naïve Bayesian filter (NBF) is proposed utilizing partially homomorphic encryption (PHE) scheme. SNBF can be implemented with only the additive homomorphism from the Paillier system, and we derive new techniques to reduce the computational cost of PHE-based SNBF. In the experiment, we implemented SNBF both in software and hardware. Compared to the best existing PHE scheme, we achieved 1,200x (resp., 398,840x) runtime reduction in the CPU (resp., ASIC) implementations, with additional 1,919x power reduction on the designated hardware multiplier. Our hardware implementation is able to classify an average-length email in 0.5 s, making it one of the most practical NBF schemes to date.

The (ring) learning with errors (RLWE/LWE) problem is one of the most promising candidates for constructing quantum-secure key exchange protocols. In this work, we design and implement specialized hardware multiplier units for both LWE and RLWE key exchange schemes to maximize their computational efficiency. By exploiting the algebraic structure with aggressive parameter sets, we show that the design and implementation of LWE key exchange on hardware is considerably easier and more flexible than RLWE. Using the proposed architectures, we show that client-side energy-efficiency of LWE-based key exchange can be on the same order, or even (slightly) better than RLWE-based schemes, making LWE an attractive option for designing post-quantum cryptographic suite.

Packed additive homomorphic encryption (PAHE) based secure neural network inference is attracting increasing attention in the field of applied cryptography. In this work, we seek to improve the practicality of LWE-based secure inference by dynamically changing the cryptographic parameters depending on the underlying architecture of the neural network. First, we develop and apply theoretical methods to closely examine the error behavior of secure inference, and propose parameters that can reduce as much as 67% of ciphertext size when smaller networks are used. Second, we use rare-event simulation techniques based on the sigma-scale sampling method to provide tight bounds on the size of cumulative errors drawn from (somewhat) arbitrary distributions. Finally, in the experiment, we instantiate an example PAHE scheme and show that we can further reduce the ciphertext size by 3.3x if we adopt a binarized neural network architecture, along with a computation speedup of 2x-3x.

The effect of negative bias temperature instability (NBTI) varies significantly according to given workloads, and thus path delay degradation is strongly dependent on each use case. In this paper, we propose a subset simulation (SS) framework that efficiently and accurately finds the worst case workload and the failure probability covering various workloads. In the proposed method, workloads that yield worst aged delay are efficiently generated by the NBTI-aware Markov chain Monte Carlo method. Through numerical experiments using benchmark circuits, the proposed method achieves up to 36 times speedup compared to the naive Monte Carlo method. From the result of the SS, feasible workload that gives worst aged delay is obtained.

The Learning with Errors (LWE) problem is a novel foundation of a variety of cryptographic applications, including quantumly-secure public-key encryption, digital signature, and fully homomorphic encryption. In this work, we propose an approximate decryption technique for LWE-based cryptosystems. Based on the fact that the decryption process for such systems is inherently approximate, we apply hardware-based approximate computing techniques. Rigorous experiments have shown that the proposed technique simultaneously achieved 1.3x (resp., 2.5x) speed increase, 2.06x (resp., 7.89x) area reduction, 20.5% (resp., 4x) of power reduction, and an average of 27.1% (resp., 65.6%) ciphertext size reduction for public-key encryption scheme (resp., a state-of-the-art fully homomorphic encryption scheme).

We propose an efficient Ising processor with approximated parallel tempering (IPAPT) implemented on an FPGA. Hardware-friendly approximations of the components of parallel tempering (PT) are proposed to enhance solution quality with low hardware overhead. Multiple replicas of Ising states having different temperatures run in parallel by sharing a single network structure, and the replicas are exchanged based on the approximated energy evaluation. The application of PT substantially improves the quality of optimization solutions. The experimental results on the various max-cut problems have shown that utilization of PT significantly increases the probability of obtaining optimal solutions, and IPAPT obtains optimal solutions two orders magnitude faster than a software solver.

A device-aging simulation that efficiently estimates temporal degradation of failure probability of a circuit is proposed. As the size of transistors shrinks, consideration of device aging in addition to manufacturing variability has become an urgent issue for maintaining reliability of LSIs. Contrary to existing techniques that separately handle manufacturing variability and the device aging, we propose a simultaneous evaluation approach using an augmented reliability and subset simulation. By eliminating the repetitive failure-probability calculations at each device-age, the proposed method reduces the number of required circuit simulations to about 1/6 of that of the conventional method without compromising accuracy.

As the transistor process technology continues to scale, the aging effect posits new challenges to the already complex static timing analysis (STA) process. In this paper, we first observe that aging can be thought of a type of correlated dynamic on-chip variations (OCV), and identify the problem introduced by such type of OCV. In particular, we take the negative bias temperature instability (NBTI) as an example dynamic OCV mechanism. We then propose a learning-based STA (LSTA) library to "predict" the timing of gates by capturing the correlation between our designed predictors. In the experiment, we used a linear regressor, support vector regression, and a non-linear method, random forest, to create the prediction model. An ISCAS'89 benchmark circuit is used as a training sample to for the algorithms to learn the aging model of gates, and the accuracies of the model is then tested on two processor-scale designs using the library are evaluated, achieving a maximum absolute error of 3.42%.

We propose an implementation of a secured content addressable memory (SCAM) based on homomorphic encryption (HE), where HE is used to compute the word matching function without the processor knowing what is being searched and the result of matching. By exploiting the shallow logic structure (XNOR followed by AND) of content addressable memory (CAM), we show that SCAM can be implemented with only additive homomorphism, greatly improving the efficiency of the HE algorithm. In the proposed method, the logic of homomorphic XNOR-AND is replaced with homomorphic XOR-OR, requiring only simple additions to be performed on the ciphertext. We also show that our scheme can be implemented by highly parallelizable and simple hardware architecture. Through experiment, we demonstrate that our software implementation is already 403x faster than the fastest known algorithm. With the help of hardware, we can achieve an energy reduction per word match by a factor of 477 million times, making our SCAM scheme much more practical.

Negative bias temperature instability (NBTI) has become one of the major reliability concerns for nanoscale CMOS technology. The NBTI effect degrades pMOS transistors by stressing them with negatively biased voltage, while the transistors heal themselves as the negative bias is removed. In this paper, we propose a cross-layer mitigation technique for NBTI-induced timing degradation in processors. The NOP (No Operation) instruction is replaced by a custom NOP instruction for healing purpose. Cells that are likely to be stressed under negative bias are classified and their upstream cell will be replaced by the internal node control (INC) logics. Upon encountering a custom NOP instruction, the INC logics will force the NBTI-stressed cell to be in its healing mode. The optimal INC logic insertion through genetic programming approach achieves much greater delay mitigation of 44.3% than prior works in a 10-year span with less than 4% of power and negligible area overhead.

As technology further scales semiconductor devices, aging-induced device degradation has become one of the major threats to device reliability. In addition, aging mechanisms like the negative bias temperature instability (NBTI) is known to be sensitive to workload (i.e., signal probability) that is hard to be assumed at design phase. In this work, we analyze the workload dependence of NBTI degradation using a processor, and propose a novel technique to estimate the worst-case paths. In our approach, with careful examination, we exploit the fact that the deterministic nature of circuit structure limits the amount of NBTI degradation on different paths, and proposes a two-stage path extraction algorithm to identify the invariable critical paths in the processor. Through numerical experiment on a MIPS32 processor, we performed a detailed signal probability analysis, and successfully extracted 85 invariable critical paths out of the 24,978 path candidates, achieving nearly 300x reduction in the sheer number of paths.

This paper proposes RTN-PUF, a novel PUF that utilizes random telegraph noise (RTN) of transistors as the physical uniqueness of individual devices. Our proposed RTN-PUF generates a response from a pair of ring oscillators (ROs) by comparing the numbers of frequency changes, which depend on the time constants of RTN. Due to the log-uniform distribution of the time constants, our RTN-PUF provides more stable responses than the existing manufacturing-variation-based PUFs. The numerical experiments show that the RTN-PUF reduces false negative errors by about 60 times compared to the conventional RO-based PUF. This facilitates to implement PUF into security purposes.

Due to latest advances in semiconductor integration, systems are becoming more susceptible to faults leading to temporary or permanent failures. We propose a new architecture extension suitable for arrays of functional units (FUs), that will provide testing and replacement of faulty units, without interrupting normal system operation. The extension relies on data-path switching realized by the proposed hot-swapping algorithm and structures, by use of which functional units are tested and replaced by spares, at lower overheads than traditional modular redundancy. For a case study architecture, hot-swapping support could be added with only 29% area overhead. In this paper we focus on experimental evaluation of the hot-swapping system from a fabricated chip in 65nm CMOS process. Autonomous testing of the hot-swapping system is enhanced with back-bias circuitry to attain an early fault detection and restoration system. Experimental measurements prove that the proposed concept works well, predicting fault occurrence with a configurable prediction interval, while power measurements reveal that with only 20% power overhead the proposed system can attain reliability levels similar to triple modular redundancy. Additionally, measurements reveal that manufacturing randomness across the die can significantly influence identical sub-circuit reliability located in different parts in the die, although identical layout has been employed.

This paper proposes a coarse-grained dynamically reconfigurable architecture, which offers flexible reliability to soft errors and aging. A notion of cluster is introduced as a basic element of the proposed architecture, each of which can select four operation modes with different levels of spatial redundancy and area-efficiency. Evaluation of permanent error rates demonstrates that four different reliability levels can be achieved by the proposed architecture. We also evaluate aging effect due to NBTI, and illustrate that alternating active cells with resting ones periodically will greatly mitigate the aging process with negligible power overhead. The area of additional circuits to attain immunity to soft errors and reliability configuration is 26.6% of the proposed reconfigurable device. Finally, a fault-tolerance evaluation of Viterbi decoder mapped on the proposed architecture suggests that there is a considerable trade-off between reliability and area overhead.

Due to latest advances in semiconductor integration, systems are becoming more susceptible to faults leading to temporary or permanent failures. We propose a new architecture extension suitable for arrays of functional units, that will provide testing and replacement of faulty units, without interrupting normal system operation. The extension relies on data path switching controlled by a hot-swapping algorithm, by use of which functional units are tested and replaced by spares if necessary, ensuring permanent operation while the spares last. The Hot-Swapping functionality could be added as a case study on a sample architecture, with an overhead of 74-87%. Also, a prototype chip (4.2 mm times 2.1 mm) targeted to operate at 166 MHz has been fabricated, using 65 nm process technology.

Emerging high-speed mobile networks enable us to receive media streaming data by mobile terminals on demand. However, media streaming with fixed bitrate causes some problems such as annoying artificial noise, interruption, and so forth, since link quality fluctuates dramatically in such mobile networks. Motivated by this, in this paper, we propose a novel media streaming system with dynamic rate control scheme to achieve continuous streaming on such unstable networks. The proposed rate control scheme uses the delay of the transcoding process to estimate network bandwidth, and adjusts the bitrate of media streams dynamically. To demonstrate our system and scheme, the proposed streaming system is evaluated under some typical models of bandwidth change in high-speed mobile networks. The results show that stable and high-quality streaming without interruption can be achieved by the proposed scheme. Moreover, we successfully demonstrate that our system performs continuous streaming on real mobile networks even in the severe network condition such as on a moving train.

This paper proposes a hardware architecture for object detection based on an AdaBoost learning algorithm with Haar-like features as weak classifiers. We analyze and discuss the parallelism in this detection algorithm and propose a partially parallel execution model suitable for hardware implementation. This parallel execution model exploits the cascade structure of classifiers, in which classifiers located near the beginning of the cascade are used more frequently than subsequent classifiers. We assign more resources to these earlier classifiers to execute in parallel than to subsequent classifiers. This dramatically improves the total processing speed without a great increase in circuit area. Moreover, the partially parallel execution model achieves flexible processing performance by adjusting the balance of parallel processing. In addition, we implement the proposed architecture on a Virtex-5 FPGA to show that it achieves real-time object detection at 30 fps on VGA video without candidate extraction.

Robust and rapid object detection is one of the great challenges in the field of computer vision. This paper proposes a hardware architecture suitable for object detection by Viola and Jones based on an AdaBoost learning algorithm with Haar-like features as weak classifiers. Our architecture realizes rapid and robust detection with two major features: hybrid parallel execution and an image scaling method. The first exploits the cascade structure of classifiers, in which classifiers located near the beginning of the cascade are used more frequently than subsequent classifiers. We assign more resources to the former classifiers to execute in parallel than subsequent classifiers. This dramatically improves the total processing speed without a great increase in circuit area. The second feature is a method of scaling input images instead of scaling classifiers. This increases the efficiency of hardware implementation while retaining a high detection rate. In addition we implement the proposed architecture on a Virtex-5 FPGA to show that it achieves real-time object detection at 30 frames per second on VGA video.