Processing-In-Memory (PIM) is an attractive solution for mitigating frequent and large data movement between computational units and memory devices. Among various PIM implementations, Samsung Aquabolt-XL is an HBM2 memory device which implements 16 PIM-enabled pseudo-channels and associates an In-Memory Processor (IMP) to each pair of the memory banks. Recent studies have shown that Aquabolt-XL can greatly accelerate various applications (e.g., deep learning) by offloading memory-intensive operations (e.g., matrix-vector multiplications) to the IMPs. However, the prior study fails to fully utilize Aquabolt-XL and achieves limited performance gains by offloading operations to the IMPs of only a single pseudo-channel. Ideally, utilizing all the 16 pseudo-channels of Aquabolt-XL can further accelerate the key operations by a factor of 16× compared to utilizing only a single pseudo-channel. To fully exploit Aquabolt-XL, therefore, memory-intensive operations should be offloaded to and concurrently executed on the IMPs of all the PIM-enabled pseudo-channels. This paper presents MPC-Wrapper, a multi-pseudo-channel wrapper interface which allows memory-intensive operations to be offloaded to and concurrently executed on the IMPs of all the 16 PIM-enabled pseudo-channels of Aquabolt-XL. First, MPC-Wrapper allows all the PIM-enabled pseudo-channels to operate independently and in parallel, thus achieving high scalability needed for fully utilizing all the PIM-enabled pseudo-channels of Aquabolt-XL. Second, MPC-Wrapper is highly flexible as it exposes the PIM-enabled pseudo-channels as separate ports and enables an FPGA logic to flexibly utilize any set of the PIM-enabled pseudo-channels according to its needs. Third, MPC-Wrapper achieves high usability by hiding the complex low-level interactions between the memory controller and Aquabolt-XL for initializing and invoking the PIM-enabled pseudo-channels from the other FPGA logics. Using an Aquabolt-XL-equipped Xilinx Alveo U280 FPGA and four memory-intensive benchmarks, we show that utilizing all the 16 PIM-enabled pseudo-channels of Aquabolt-XL with MPC-Wrapper achieves a geometric mean speedup of 13.66× over the baseline single PIM-enabled pseudo-channel implementations of the benchmarks.

The imminent rise in data consumption and the physical constraints of current advanced CMOS scaling hasten the end of the projection to the binary system. For a breakthrough of these issues, the ternary system, known for its superior efficiency in expressing numbers (closest to$\mathrm {e}\approx ~2.7183$) has garnered considerable attention. Among the ternary studies reported, the anti-ambipolar transistor (AAT) is acquiring attention thanks to its unique negative differential resistance (NDR) and anti-ambipolar characteristics (AAC). Moreover, easy-to-fabricate inkjet-printing based AAT was introduced. Therefore, in this paper, we propose a practical design methodology (‘$\mathbf {T^{3}L:} $The Tri-transistor Ternary Logic’) and a set of novel ternary logic based on inkjet-printed AATs and CMOSs. In detail, 1) We propose balanced ternary full adders (BTFA) and prove that inkjet-printed AATs and CMOSs are highly capable of implementing any kind of ternary logic. 2) We propose two design methodologies for ternary logic design: NDR-based Design Method I and AAC-based Design Method II. 3) We present optimization methodology for inkjet-printed ternary circuit stability and provide circuitry to secure sufficient noise margin. We provide a highly-compact BTFA design that requires only 64 transistors and an ultra-low-power BTFA design that reduces power by 84.7% to 98.8% compared to the previous designs.

Serverless computing like Function-as-a-Service (FaaS) is attractive for IoT service providers, liberating the providers from server maintenance. Since a data processing function is executed on the cloud instead of a dedicated server in the FaaS platform, the service users send their private data in their IoT devices to the third-party cloud, taking privacy leakage risks. Homomorphic encryption (HE) can preserve the privacy by enabling encrypted data processing on the cloud, but using HE for every data item incurs large computation and communication overheads. This work proposes SelectiveCrypt, a compiler-assisted semantic-aware encryption scheme that applies different cryptographic primitives depending on the operations on each data item. SelectiveCrypt homomorphically encrypts data items if arithmetic operations are applied to the data, while SelectiveCrypt encrypts data items with a symmetric key if the data are stored in the cloud without any arithmetic operation. The SelectiveCrypt framework consists of a compiler and its runtime system. The SelectiveCrypt compiler statically analyzes the data processing, determines an appropriate cryptographic primitive for each data item, and automatically transforms arithmetic operations into the homomorphic computation. The SelectiveCrypt runtime encrypts and decrypts the data items according to the static analysis result. This work evaluates the prototype SelectiveCrypt framework with five benchmarks that reflect real-world IoT scenarios. The evaluation results show that the SelectiveCrypt framework successfully reduces response time and communication overhead by 1.59 times and 9.61 times, respectively, compared with a HE scheme.

The Internet of Things (IoT) enables new services, such as smart home and smart healthcare integrating various resources of networked devices. However, managing IoT resources is very challenging because a device may have multiple different resources and the service accesses subsets of the resources in multiple devices. To simplify the resource management, the existing IoT frameworks provide resource models and protocols for manufacturers and programmers, but their access granularity is too coarse grained causing overprivilege problems, or too fine grained causing management overheads. To avoid the overprivilege problems without the management overheads, this work proposes a new composable and flexible resource management scheme and implements its prototype compiler-runtime framework called ComFlex. The ComFlex compiler allows manufacturers to register their devices as composition of fine-grained resources by inheriting the existing interfaces of the resources, and programmers to define their own access granularity as a virtual resource that consists of the fine-grained resources of multiple devices. The ComFlex runtime supports fine-grained access control without additional overheads by mapping the fine-grained resources of the virtual resource into physical resources of different devices. To evaluate the ComFlex framework, this work implements 52 resources on 14 devices and six IoT services with ComFlex and an existing fine-grained resource management scheme. Compared to the existing scheme, the ComFlex framework supports the IoT services with 40.2% fewer lines of code and 80.9% less discovery time, without any overprivilege problem and response time delay.

In the Internet of Things (IoT) environment, detecting a faulty device is crucial to guarantee the reliable execution of IoT services. To detect a faulty device, existing schemes trace a series of events among IoT devices within a certain time window, extract correlations among them, and find a faulty device that violates the correlations. However, if a few users share the same IoT environment, since their concurrent activities make non-correlated devices react together in the same time window, the existing schemes fail to detect a faulty device without differentiating the concurrent activities. To correctly detect a faulty device in the multiple concurrent activities, this work proposes a new precise correlation extraction scheme, called PCoExtractor. Instead of using a time window, PCoExtractor continuously traces the events, removes unrelated device statuses that inconsistently react for the same activity, and constructs fine-grained correlations. Moreover, to increase the detection precision, this work newly defines a fine-grained correlation representation that reflects not only sensor values and functionalities of actuators but also their transitions and program states such as contexts. Compared to existing schemes, PCoExtractor detects and identifies 40.06% more faults for 4 IoT services with concurrent activities of 12 users while reducing 80.3% of detection and identification times.

In the Internet-of-Things (IoT) environments, users define event-condition-action (ECA) rules, and expect IoT frameworks to evaluate conditions and take appropriate actions within a certain time limit after an event occurs. To evaluate the conditions with fresh data items, the frameworks acquire required data from IoT sensors. Since the data acquisition causes battery consumption of sensors, the frameworks should minimize the number of the data acquisition while keeping the sensor data fresh until finishing the condition evaluation. However, existing data acquisition schedulers inefficiently acquire sensor data because the schedulers assume each ECA rule in a program is independent of each other although different rules may share some sensing data from the same sensors. This work proposes an efficient sharing-aware data acquisition scheduling algorithm that reduces unnecessary data acquisition by sharing sensor data commonly used in different rules while satisfying time constraints. To evaluate the proposed scheduling algorithm, this work deploys 19 devices in an office, collects values of 26 different sensors for 144 hours, and simulates the proposed algorithm and a baseline algorithm. Compared to the baseline algorithm, the proposed algorithm reduces communication count and deadline miss ratio by 31.9% and 50.2% respectively.

In the Internet of Things (IoT) environments, cloud servers integrate various IoT devices including sensors and actuators, and provide new services that assist daily lives of users interacting with the physical world. While response time is a crucial factor of quality of the services, supporting short response time is challenging for the cloud servers due to a growing number and amount of connected devices and their communication. To reduce the burden of the cloud servers, fog computing is a promising alternative to offload computation and communication overheads from the cloud servers to fog nodes. However, since existing fog computing frameworks do not extract codes for fog nodes fully automatically, programmers should manually write and analyze their applications for fog computing. This work proposes Spinal Code, a new compiler-runtime framework for near-user computation that automatically partitions an original cloud-centric program into distributed sub-programs running over the cloud and fog nodes. Moreover, to reduce response time in the physical world, Spinal Code allows programmers to annotate latency sensitive actuators in a program, and optimizes the critical paths from required sensors to the actuators when it generates the sub-programs. This work implements 9 IoT programs across 4 service domains: healthcare, smart home, smart building and smart factory, and demonstrates that Spinal Code successfully reduces 44.3% of response time and 79.9% of communication on the cloud compared with a cloud-centric model.

Content-based audio retrieval including query-by-example (QBE) and query-by-vocal imitation (QBV) is useful when search-relevant text labels for the audio are unavailable, or text labels do not sufficiently narrow the search. However, a single query example may not provide sufficient information to ensure the target sound(s) in the database are the most highly ranked. In this paper, we adapt an existing model for generating audio embeddings to create a state-of-the-art similarity measure for audio QBE and QBV. We then propose a new method to update search results when top-ranked items are not relevant: The user provides an additional vocal imitation to illustrate what they do or do not want in the search results. This imitation may either be of some portion of the initial query example, or of a top-ranked (but incorrect) search result. Results show that adding vocal imitation feedback improves initial retrieval results by a statistically significant amount.

Tagging of environment audio events is essential in many areas. However, finding sound events and labeling them within a long audio file is tedious and time-consuming. Building an automatic recognition system using modern machine learning is often not feasible because it requires a large number of human-labeled training examples and it is not reliable enough for all uses. I propose interactive sound event detection to solve the issue by combining machine search with human tagging, specifically focusing on the effectiveness of various types of user-inputs to the interactive sound searching. The types of user inputs that I will explore include binary relevance feedback, segmentation, and vocal imitation. I expect that leveraging one or combination of these user inputs would help users find audio contents of interest quickly and accurately, even in the situation where there are not enough training examples for a typical automated system.

Labeling of audio events is essential for many tasks. However, finding sound events and labeling them within a long audio file is tedious and time-consuming. In cases where there is very little labeled data (e.g., a single labeled example), it is often not feasible to train an automatic labeler because many techniques (e.g., deep learning) require a large number of human-labeled training examples. Also, fully automated labeling may not show sufficient agreement with human labeling for many uses. To solve this issue, we present a human-in-the-loop sound labeling system that helps a user quickly label target sound events in a long audio. It lets a user reduce the time required to label a long audio file (e.g., 20 hours) containing target sounds that are sparsely distributed throughout the recording (10% or less of the audio contains the target) when there are too few labeled examples (e.g., one) to train a state-of-the-art machine audio labeling system. To evaluate the effectiveness of our tool, we performed a human-subject study. The results show that it helped participants label target sound events twice as fast as labeling them manually. In addition to measuring the overall performance of the proposed system, we also measure interaction overhead and machine accuracy, which are two key factors that determine the overall performance. The analysis shows that an ideal interface that does not have interaction overhead at all could speed labeling by as much as a factor of four.

Tagging of sound events is essential in many research areas. However, finding sound events and labeling them within a long audio file is tedious and time-consuming. Building an automatic recognition system using machine learning techniques is often not feasible because it requires a large number of human-labeled training examples and fine tuning the model for a specific application. Fully automated labeling is also not reliable enough for all uses. We present I-SED, an interactive sound detection interface using a human-in-the-loop approach that lets a user reduce the time required to label audio that is tediously long (e.g. 20 hours) to do manually and has too few prior labeled examples (e.g. one) to train a state-of-the-art machine audio labeling system. We performed a human-subject study to validate its effectiveness and the results showed that our tool helped participants label all target sound events within a recording twice as fast as labeling them manually.

The explosion of networked devices has driven a new computing environment called the Internet of Things (IoT), enabling various services such as home automation and health monitoring. Despite the promising applicability of the IoT, developing an IoT service is challenging for programmers, because the programmers should integrate multiple programmable devices and heterogeneous third-party devices. Recent works have proposed integrated programming platforms, but they either require device-specific implementation for third-party devices without any device abstraction, or abstract all the devices to the standard interfaces requiring unnecessary abstraction of programmable devices. To integrate IoT devices with selective abstraction, this work revisits the object oriented programming (OOP) model, and proposes a new language extension and its compiler-runtime framework, called Esperanto. With three annotations that map each object to its corresponding IoT device, the Esperanto language allows programmers to integrate multiple programmable devices into one OOP program and to abstract similar third-party devices into their common ancestor classes. Given the annotations, the Esperanto compiler automatically partitions the integrated program into multiple sub-programs for each programmable IoT device, and inserts communication and synchronization code. Moreover, for the ancestor classes, the Esperanto runtime dynamically identifies connected third-party devices, and links their corresponding descendent objects. Compared to an existing approach on the integrated IoT programming, Esperanto requires 33.3% fewer lines of code to implement 5 IoT services, and reduces their response time by 44.8% on average.

Integrating various networked devices, the Internet of Things (IoT) enables various new services like home automation, making its market larger and more competitive. Although rapid development of an IoT application is crucial to keep up with the highly competitive IoT market, developing an IoT application is challenging for programmers because the programmers should integrate multiple programmable devices and heterogeneous third-party devices. Some IoT frameworks integrate programming environments of multiple devices, but they either require device-specific implementation for third-party devices without any device abstraction, or abstract all the devices to the standard interfaces requiring unnecessary abstraction of programmable devices. This work introduces the Esperanto framework that integrates IoT devices with selective abstraction, allowing rapid prototyping of an IoT application. Exploiting the correspondence between an object and a thing in the object oriented programming (OOP) model, the Esperanto framework allows programmers to write only one OOP program instead of multiple programs for each device, and to manipulate third-party devices with their common ancestor classes. Compared to an existing approach on the integrated IoT programming, Esperanto requires 33.3% fewer lines of code to implement 5 IoT services, and reduces their response time by 44.8% on average. Moreover, with an empirical study, this work shows that the Esperanto framework reduces the development time by 52.7%.

Recommending media objects to users typically requires users to rate existing media objects so as to understand their preferences. The number of ratings required to produce good suggestions can be reduced through collaborative filtering. Collaborative filtering is more difficult when prior users have not rated the same set of media objects as the current user or each other. In this work, we describe an approach to applying prior user data in a way that does not require users to rate the same media objects and that does not require imputation (estimation) of prior user ratings of objects they have not rated. This approach is applied to the problem of finding good equalizer settings for music audio and is shown to greatly reduce the number of ratings the current user must make to find a good equalization setting.

Audio equalizers (EQs) are perhaps the most commonly used tools used in audio production. The SocialEQ project is a web-based personalized audio equalization system that uses an alternative interface paradigm to the standard approach. Here, the user names a desired effect (e.g. Make the sound "warm") and teaches the tool (e.g. An equalizer) what settings make the sound embody the term. Social EQ typically requires 25 ratings to properly personalize the equalization settings. In this paper, we present three methods to improve the speed of generating personalized items (audio settings) so users can be provided personalized EQ curves after rating a much smaller number of examples. These methods can be adapted to any situation where collaborative filtering is desirable, the end products created for users are unique and comparable to each other, but prior users did not rate the same set of examples as the current user. Methods are tested on a data set of 1635 user sessions.