We propose an approach to leverage multi-bounce returns of a flash LiDAR on portable smartphones for 3D specular surface reconstruction. Traditional LiDAR systems assume that all returns are one-bounce returns, which can lead to an overestimation of the true mirror surface and cause it to appear as if there is a hole. However, in reality, returns from mirror surfaces follow multi-bounce paths. We operate with a consumer-grade, coarse multi-beam flash LiDAR, enabling real-time mapping on an affordable and portable smartphone. To address the challenges posed by the coarse setup, where the transmitter and receiver are co-located, we propose solving the association problem using the ‘reciprocal pair’ algorithm. This algorithm can distinguish between different types of bounces from multi-bounce returns. We have demonstrated detection over multiple consecutive frames for dense mirror mapping. In addition to 3D reconstruction, we show that multi-bounce returns enhance performance in applications such as segmentation and novel view synthesis. Our method can be integrated with state-of-the-art learned-based models, enhancing their robustness in discerning ambiguous scenarios. Importantly, our approach can map various specular surfaces like mirrors and glasses without assuming specific shapes, and it can operate on non-perpendicular specular-diffuse surface pairs.

General Matrix Multiply (GEMM) units, consisting of multiply-accumulate (MAC) arrays, perform bulk of the computation in deep learning (DL). Recent work has proposed a novel MAC design, Bit-Pragmatic (PRA), capable of dynamically exploiting bit sparsity. This work presents OzMAC (Omit-zero-MAC), a modified re-implementation of PRA, but extends beyond earlier works by performing rigorous post-synthesis evaluation against binary MAC design across multiple bitwidths and clock frequencies using TSMC N5 process node to assess commercial implementation potential. We demonstrate the existence of high bit sparsity in eight pretrained INT8 DL workloads and show that 8-bit OzMAC improves all three metrics of area, power, and energy significantly by 21%, 70%, and 28%, respectively. Similar improvements are achieved when scaling data precisions (4, 8, 16 bits) and clock frequencies (0.5 GHz, 1 GHz, 1.5 GHz). For the 8-bit OzMAC, scaling its frequency to normalize the throughput, it still achieves 30% improvement on both power and energy.

Self-driving vehicles (SDVs) hold great potential for improving traffic safety and are poised to positively affect the quality of life of millions of people. To unlock this potential one of the critical aspects of the autonomous technology is understanding and predicting future movement of vehicles surrounding the SDV. This work presents a deep-learning- based method for kinematically feasible motion prediction of such traffic actors. Previous work did not explicitly encode vehicle kinematics and instead relied on the models to learn the constraints directly from the data, potentially resulting in kinematically infeasible, suboptimal trajectory predictions. To address this issue we propose a method that seamlessly combines ideas from the AI with physically grounded vehicle motion models. In this way we employ best of the both worlds, coupling powerful learning models with strong feasibility guarantees for their outputs. The proposed approach is general, being applicable to any type of learning method. Extensive experiments using deep convnets on real-world data strongly indicate its benefits, outperforming the existing state-of-the-art.

Following detection and tracking of traffic actors, prediction of their future motion is the next critical component of a self-driving vehicle (SDV) technology, allowing the SDV to operate safely and efficiently in its environment. This is particularly important when it comes to vulnerable road users (VRUs), such as pedestrians and bicyclists. These actors need to be handled with special care due to an increased risk of injury, as well as the fact that their behavior is less predictable than that of motorized actors. To address this issue, in the current study we present a deep learning-based method for predicting VRU movement, where we rasterize high-definition maps and actor's surroundings into a bird's-eye view image used as an input to deep convolutional networks. In addition, we propose a fast architecture suitable for real-time inference, and perform an ablation study of various rasterization approaches to find the optimal choice for accurate prediction. The results strongly indicate benefits of using the proposed approach for motion prediction of VRUs, both in terms of accuracy and latency.

We address one of the crucial aspects necessary for safe and efficient operations of autonomous vehicles, namely predicting future state of traffic actors in the autonomous vehicle's surroundings. We introduce a deep learning-based approach that takes into account a current world state and produces raster images of each actor's vicinity. The rasters are then used as inputs to deep convolutional models to infer future movement of actors while also accounting for and capturing inherent uncertainty of the prediction task. Extensive experiments on real-world data strongly suggest benefits of the proposed approach. Moreover, following successful tests the system was deployed to a fleet of autonomous vehicles.

Autonomous driving presents one of the largest problems that the robotics and artificial intelligence communities are facing at the moment, both in terms of difficulty and potential societal impact. Self-driving vehicles (SDVs) are expected to prevent road accidents and save millions of lives while improving the livelihood and life quality of many more. However, despite large interest and a number of industry players working in the autonomous domain, there still remains more to be done in order to develop a system capable of operating at a level comparable to best human drivers. One reason for this is high uncertainty of traffic behavior and large number of situations that an SDV may encounter on the roads, making it very difficult to create a fully generalizable system. To ensure safe and efficient operations, an autonomous vehicle is required to account for this uncertainty and to anticipate a multitude of possible behaviors of traffic actors in its surrounding. We address this critical problem and present a method to predict multiple possible trajectories of actors while also estimating their probabilities. The method encodes each actor's surrounding context into a raster image, used as input by deep convolutional networks to automatically derive relevant features for the task. Following extensive offline evaluation and comparison to state-of-the-art baselines, the method was successfully tested on SDVs in closed-course tests.

We present a compiler for Loihi, a novel manycore neuromorphic processor that features a programmable, on-chip learning engine for training and executing spiking neural networks (SNNs). An SNN is distinguished from other neural networks in that (1) its independent computing units, or "neurons", communicate with others only through spike messages; and (2) each neuron evaluates local learning rules, which are functions of spike arrival and departure timings, to modify its local state. The collective neuronal state dynamics of an SNN form a nonlinear dynamical system that can be cast as an unconventional model of computation. To realize such an SNN on Loihi requires each constituent neuron to locally store and independently update its own spike timing information. However, each Loihi core has limited resources for this purpose and these must be shared by neurons assigned to the same core. In this work, we present a compiler for Loihi that maps the neurons of an SNN onto and across Loihi's cores efficiently. We show that a poor neuron-to-core mapping can incur significant energy costs and address this with a greedy algorithm that compiles SNNs onto Loihi in a power-efficient manner. In so doing, we highlight the need for further development of compilers for this new, emerging class of architectures.

Orthogonal matching pursuit (OMP) is an efficient approximation algorithm for computing sparse representations. However, prior research has shown that the representations computed by OMP may be of inferior quality, as they deliver suboptimal classification accuracy on several im- age datasets. We have found that this problem is caused by OMP’s relatively weak stability under data variations, which leads to unreliability in supervised classifier training. We show that by imposing a simple nonnegativity constraint, this nonnegative variant of OMP (NOMP) can mitigate OMP’s stability issue and is resistant to noise overfitting. In this work, we provide extensive analysis and experimental results to examine and validate the stability advantage of NOMP. In our experiments, we use a multi-layer deep architecture for representation learning, where we use K-means for feature learning and NOMP for representation encoding. The resulting learning framework is not only efficient and scalable to large feature dictionaries, but also is robust against input noise. This framework achieves the state-of-the-art accuracy on the STL-10 dataset.

This paper introduces an approach to accomplish motion segmentation from a moving stereo camera based on deep learning. Previous work on moving object detection mostly use point features based on 3D geometric constraints. However, point features require good features, and are hard to detect or to be matched correctly in situations where objects have smooth textures. To alleviate this problem, learning high-level spatio-temporal features unsupervisedly from raw image data based on Reconstruction Independent Component Analysis (RICA) autoencoders is proposed. Despite the power of the new spatio-temporal features, these features cannot not learn and be used to interpret 3D geometry of dynamic scenes, which is critical for moving object detection from moving cameras. As detected moving points based on 3D geometric constraints still contain valuable information of 3D scene as well as the camera egomotion, we propose a framework that incorporates both the detected moving point results and the learned spatio-temporal features as inputs to Recursive Neural Networks (RNN) that performs motion segmentation. Both features effectively complement each other. The proposed approach is demonstrated with real-world stereo video data that contains multiple moving objects, and has achieved 26% better detection rate over the existing 3D geometric-based moving points detector.

This paper presents MIMO/CON, a PHY/MAC cross-layer design for multiuser MIMO wireless networks that delivers throughput scalable to many users. MIMO/CON supports concurrent channel access from uncoordinated and loosely synchronized users. This new capability allows a multi-antenna MIMO access point (AP) to fully realize its MIMO capacity gain. MIMO/CON draws insight from compressive sensing to carry out concurrent channel estimation. In the MAC layer, MIMO/CON boosts channel utilization by exploiting normal MAC layer retransmissions to recover otherwise undecodable packets in a collision. MIMO/CON has been implemented and validated on a 4×4 MIMO testbed with software-defined radios. In software simulations, MIMO/CON achieves a 210% improvement in MAC throughput over existing staggered access protocols in a 5-antenna AP scenario.

In this paper, a vCPU (virtual CPU) migration mechanism in order to improve real-time responsiveness in a GPOS (General Purpose Operating System) is presented in the embedded multicore virtualization platform and can also be applied to CPS environment. In a GPOS/RTOS (Real Time Operating System) virtualization system nowadays, tasks in GPOS, however, also need some degree of real-time services from the system. Unfortunately, in a traditional virtualization platform, a virtualization layer always gives RTOS vCPU higher priority and preempts the execution of GPOS. Therefore, a kernel module in the GPOS is added in our virtualization system to export and boost a GPOS vCPU contexts which needs higher priority against RTOS vCPU. At the same time, in order not to sacrifice the performance of the victim RTOS vCPU, a vCPU migration mechanism is added to our virtualization system, and migrates the RTOS vCPU to a low-loading CPU when this vCPU is preempted. Performance improvement of GPOS's real-time responsiveness is also given while a detailed analysis of the overhead of the RTOS vCPU is performed as well.

We propose a medium access control (MAC) protocol for wireless local area networks (LANs) that leverages the theory of compressive sensing. The proposed compressive sensing MAC (CS-MAC) exploits the sparse property that, at a given time, only a few hosts are expected to request for radio channel access. Under CS-MAC, a central coordinator, such as a wireless access point (AP) can recover a multitude of these requests in one decoding operation, and then schedule multiple hosts accordingly. The coordinator is only required to receive a relatively small number of random projections of host requests, rather than polling individual hosts. This results in an efficient request-grant method. Via a hardware prototype based on a software-defined radio platform, we demonstrate the feasibility of realizing CS-MAC with compressive measurements formed in the air to achieve high efficiency.

In this paper, we introduce a lightweight processor abstraction layer named SPUMONE. SPUMONE provides virtual CPUs for respective guest OSes, and schedules them according to their priorities. In a typical case, SPUMONE schedules Linux with a low priority and an RTOS with a high priority. We first discuss why the traditional virtual machine monitor design is not appropriate for embedded systems, and how the features of SPUMONE allow us to design modern complex embedded systems with less efforts. Then, we describe two features of SPUMONE for the real-time resource management. SPUMONE also offers a novel mechanism to protect a critical component from malicious programs injected into the GPOS kernel.

When a real-time OS (RTOS) and a symmetric multiprocessing general purpose OS (SMP GPOS) are executed on the same multicore processor, there is a possibility to cause the lock holder preemption (LHP) problem, which is a well known problem of the sources of the performance degradation on an SMP GPOS. The problem occurs when a kernel mode thread holding a lock in a SMP GPOS is preempted by other OSes if these OSes share the same physical core. The existing solution does not take into account real-time responsiveness so it is not suitable to use in embedded systems. The paper proposes two new techniques to avoid the LHP problem for ensuring both the real-time responsiveness of real-time OS and the high throughput of SMP GPOS. The new proposed techniques have been implemented on our virtualization layer called SPUMONE, and we measured the results showing that the proposed new techniques reduce the LHP problem without degrading the real-time responsiveness.

A virtualization layer makes it possible to compose multiple functionalities on a multi-core processor with minimum modifications of OS kernels and applications. A multi-core processor is a good candidate to compose various software independently developed for dedicated processors into one multi-core processor to reduce both the hardware and development cost. In this paper, we present SPUMONE, which is a virtualization layer suitable for developing multi-core processor based-information appliances.

Wireless data transfer under high mobility, as found in unmanned aerial vehicle (UAV) applications, is a challenge due to varying channel quality and extended link outages. We present FlowCode, an easily deployable link-layer solution utilizing multiple transmitters and receivers for the purpose of supporting existing transport protocols such as TCP in these scenarios. By using multiple transmitters and receivers and by exploiting the resulting antenna beam diversity and parallel transmission effects, FlowCode increases throughput and reception range. In emulation, we show that TCP over FlowCode gives greater goodput over a larger portion of the flight path, compared to an enhanced TCP protocol using the standard 802.11 MAC. In the process, we make a strong case for using trace-modulated emulation when developing distributed protocols for complex wireless environments.

In this paper, we propose a virtualization architecture for the multi-core embedded system to provide more system reliability and security while maintaining the same performance without introducing additional special hardware supports or having to implement complex protection mechanism in the virtualization layer. Virtualization has been widely used in embedded systems, especially in consumer electronics, albeit itself is not a new technique, because there are various needs for both GPOS (General Purpose Operating System) and RTOS (Real Time Operating System). The surge of the multi-core platform in the embedded system also helps the consolidation of the virtualization system for its better performance and lower power consumption. Embedded virtualization design usually uses two kinds of approaches. The first one is to use the traditional VMM, but it is too complicated for use in the embedded environment if there is no additional special hardware support. The other is the use of the micro kernel which imposes a modular design. The guest systems, however, would suffer from considerable amount of modifications because the micro kernel lets the guest systems to run in user space. For some RTOSes and theirs applications originally running in kernel space, it makes this approach more difficult to work because a lot of privileged instructions are used in those codes. To achieve better reliability and keep the virtualization layer design light weighted, a common hardware component adopted in the multi-core embedded processors is used in this work. In the most embedded platforms, vendors provide additional on-chip local memory for each physical core and these local memory areas are private only to their cores. By taking this memory architecture's advantage, we can mitigate above-mentioned problems at once. We choose to re-map the virtualization layer's program called SPUMONE, which it runs all its guest systems in kernel space, on the local memory. By doing so, it can provide additional reliability and security for the entire system because the SPUMONE's design in a multi-core platform has each instance being installed on a separated processor core which is different from the traditional virtualization layer design and the content of each SPUMONE is inaccessible to each others. We also achieve this goal without bringing any overhead to the overall performance.

In Wireless Access in Vehicular Environments (WAVE), the terminal devices are usually highly mobile vehicles or transportation tools, which handoff much more frequently than those in wireless networks. However, the quality of network services, like VoIP and multimedia streaming, will be seriously influenced by too frequent handoff or too long handoff latency. As a result, traditional handoff mechanisms no longer satisfy the need of vehicular communications. At present, most existing fast handoff mechanisms are based on the signal strength of the APs, which does not suit vehicular communications fully. Generally speaking, the regional traffic in Vehicular Ad-hoc Networks (VANETs) keep changing and varying. For example, on busy roads or in rush hours, the increase of number of vehicles may lead to the network congestion and degrade the performance of vehicular communications substantially. Therefore, this paper presents the QualityScan scheme, an efficient pre-scanning method to improve the handoff performance in VANETs. With the attempt to reduce the handoff latency and maintain the load balance among the APs, our proposed scheme regularly collects the loading states of the nodes by the pre-established AP controller (APC), and predicts the network traffic of the next moment. Consequently, Mobile Nodes (MNs) can select the most suitable AP for the best QoS according to the parameters received by passive scanning, while more QoS parameters should participate in cooperation. The simulation result proves that our proposed scheme not only maintains the QoS and the load balance among the nodes, but also further enhances the handoff performance in VANETs.

In this paper, we propose a virtualization architecture for the multi-core embedded system to provide more sys-tem reliability and security while maintaining the same performance without introducing additional special hardware supports or having to implement complex protection mechanism in the virtualization layer. Embedded virtualization design usually uses two kinds of approaches, traditional VMM and microkernel approaches, but both of them suffer from performance or engineering cost problems. To achieve better reliability and keep the virtualization layer design light weighted, a common hardware component called local memory adopted in the multi-core embedded processors is used in this work. By taking this memory ar-chitecture's advantage, we can mitigate above-mentioned problems at once. We choose to re-map the virtualizationlayer's program called SPUMONE, which it runs all its guest systems in kernel space, on the local memory. By doing so, it can provide additional reliability and security for the entire system because the SPUMONE's design in a multi-core platform has each instance being installed on a separated processor core, which is different from the traditional virtualization layer design, and therefore the content of each SPUMONE in the local memory is inaccessible to each others.

Multi-core processors are being increasingly adopted for embedded systems because they improve performance, power consumption and lower development cost. Composing multiple operating systems on a multi-core processor enhances the reusability of software when developing rich functional embedded systems. Multiple OS environments enable the product to use two versions of an operating system at the same time. In order to build multiple OS environments, a virtualization layer specialized for embedded systems is necessary, since most of processors for embedded systems support only two protection levels, and there is no hardware support for virtualization. In traditional approaches, an OS kernel runs at the user level to isolate the respective OS kernels to increase reliability, but this approach requires heavy modifications to the guest OSes if there is no proper hardware virtualization support that is rarely used in embedded systems. Therefore existing solutions are not preferred by the embedded system industry. In this paper, we propose a composition kernel where multiple OS kernels are running on top of a very thin hardware abstraction layer. A composition kernel can reduce the engineering cost of developing an embedded system by reusing existing OS kernels and application with minimum modification without assuming special hardware supports.

As the R&D experience accumulates, there is a rising interest of wireless sensor network (WSN) deployment in the urban environment. For mission critical applications such as healthcare and workplace safety, in particular, it is essential that the data dissemination mechanisms satisfy two important quality of service (QoS) requirements: (1) high delivery rate and (2) low transmission delay. Proposed in this work is a cross-layer designed data dissemination mechanism, referred to as Cross-Layer Diffusion (XD), in which notions in the path discovery (routing) component are exploited by data forwarding (MAC) component to improve the delivery rate and transmission delay. Using traces collected from a prototype WSN deployed in urban environment, we compare XD to the state-of-the-art mechanisms and find that XD is not only more efficient but also more practical.

In this work, we present an experimental study, in which we deploy a 14-node 802.15.4-based sensor network in our department building. Using the k-nearest-neighbor (KNN) method to the estimation of location, we implement an RSSI-signature-based localization system on top of the network.

We consider the problem of localizing wireless nodes in an outdoor, open-space environment, using ad-hoc radio ranging measurements, e.g., 802.11. We cast these ranging measurements as a set of distance constraints, thus forming an over-determined system of equations suitable for non-linear least squares optimization. However, ranging measurements are often subject to errors, induced by multipath signals and variations in path loss, unreliable hardware or antenna connectors, or imperfection in measurement models. Such potentially large, non-Gaussian errors in the measurement data ultimately produce inaccurate localization solutions. We propose a new error-tolerant localization method, called snap-inducing shaped residuals (SISR), to identify automatically "bad nodes" and "bad links" arising from these errors, so that they receive less weight in the localization process. In particular, SISR snaps "good nodes" to their accurate locations and gives less emphasis to other nodes. While the mathematical techniques used by SISR are similar to robust statistics, SISR's exploitation of the snap-in effect in localization appears to be novel. We provide analysis on the principle of SISR, illustrate errors in real-world measurements, and demonstrate a working SISR implementation in field experiments on a testbed of 37 wireless nodes, as well as show the superior performance of SISR in simulation with a larger number of nodes.

Boundary detection is a form of location-aware services that aims at detecting targets crossing certain critical regions. Typically, a lower location sampling rate contributes to a lower level of energy consumption but, in the meantime, delays the detection of boundary crossing events. Opting to enable energy-efficient boundary detection services, we propose a mobility-aware mechanism that adapts the location sampling rate to the target mobility. Results from our simulations and live experiments confirm that the proposed adaptive sampling mechanism is effective. In particular, when experimented with realistic errors measured from a live radio-frequency-based localization system, the energy consumption can be reduced significantly to 20 percent.

With the creative use of sensors, in the future, sensor networks could impact on a wide variety of applications from national security to consumer electronics. Sharing the vision and emphasizing the energy efficiency of data dissemination for resource inventory applications, we propose a novel data aggregation strategy and algorithms to obtain the population of the objects (i.e., the resource) to be sensed in the networks. With a set of cautiously validated simulations, we show that the energy consumption in applying our solution is only one third of the original case and scalable to the object population. Experimental results also show that we are able to obtain consistent lower and upper bounds at 70% and 180% of the exact population. While we show the solution works effectively for resource inventory applications, this work suggests more profoundly that there is yet design space to explore in application-specific data aggregation for energy conservation.