In this paper, we present an open-source RISC-V processor with an integrated dynamic memory manager hardware module. Traditionally, the management of the main memory of a computing system is handled by a software library. However, the process involves searching and manipulation of the link lists of memory blocks, which can be expensive when the memory becomes fragmented. As a result, for embedded systems that have to be online for a long duration, a static data structure is often used to reduce the overhead of dynamic memory management at the cost of less software flexibility. Nevertheless, modern VLSI technology allows the efficient implementations of hardwired resource managers directly into the processor microarchitecture for better performance. As the experiments in this paper show, a hardware memory manager integrated within the processor core can be much more efficient than using a software library. Hardwired resource managers are particularly useful for IOT devices since the processors typically run at a lower clock rate. The proposed architecture is implemented and verified on a Xilinx FPGA development board and will be made open source.

In this paper, we present the design of an open-source RISC-V application processor with an embedded TCP/IP network module. Traditionally, the TCP/IP stack is a software layer of the OS kernel due to its complex control behavior. However, previous studies show that a hardwired logic can perform the TCP/IP control algorithms much more efficiently than a software implementation. However, to allow a processor to invoke a hardware TCP/IP logic efficiently is not a trivial task. This paper proposes an efficient interface logic between the processor core and the hardware TCP/IP stack through user-defined RISC-V instructions. The proposed architecture is implemented and verified on a Xilinx FPGA development board. Experimental results show that the average end-to-end packet delay can be reduced by up to 99% using the proposed network module when compared against the software network stack under the FreeRTOS real-time operating system. Therefore, the proposed architecture can be very useful for deeply-embedded IOT devices where a low-power processor can be used to handle low-latency high throughput IP packet transmissions.

This paper presents the design and implementation of a hardwired thread scheduler circuit with multi-level priority queues for a four-core Java application processor. A hardwired thread scheduler is much more efficient than the software thread scheduler in a software OS kernel, such as Linux. Since the hardware scheduler can operate in parallel with the processor cores, complex scheduling decisions can be made while the processor cores are running applications. In addition, single-cycle context-switching is possible and no processor core has to waste time running the scheduler. Full-system implementation of a four-core Java processor with the hardware scheduler has been verified using a Xilinx Kintex-7 FPGA device. Performance evaluations show that the proposed system scales up very well and is promising for deeply-embedded multi-thread applications such as the automatic driver assistance systems or the drones.

This paper presents the design and implementation of a hardwired OS kernel circuitry inside a Java application processor to provide the system services that are traditionally implemented in software. The hardwired system functions in the proposed SoC include the thread manager, the memory manager, and the I/O subsystem interface. There are many advantages in making the OS kernel a hardware component, such as a fast system boot time, highly efficient single-core multi-thread context-switching performance, and a better potential for supporting a complex multi-level memory subsystem. In addition, since the target application processor used in this paper is based on a Java processor, the system is not susceptible to the stack and pointer-based security attacks that are common to the register-based processors. Full-system performance evaluations on an FPGA show that the proposed system is very promising for deeply-embedded multi-thread applications.

In this paper, we have implemented a dynamic pipeline-partitioning video decoder for the symmetric stream multiprocessor (SSMP) architecture. The SSMP architecture extends the traditional symmetric multiprocessor (SMP) architecture with dedicated per-core scratchpad memories and inter-processor communication (IPC) controllers for efficient data passing between the processor cores. The SSMP architecture allows the processor cores to cooperate efficiently in a fine-grained software pipeline fashion. A traditional software pipelined video decoder has fixed pipeline-stage partitions. The AVC/H.264 video decoder investigated in this paper dynamically assigns different stages of the video macroblock (MB) decoding tasks to different processor cores in order to maintain load balance among the processor cores. The pipeline partitioning policy is based on the queue levels of the inter-stage buffers. Experimental results show that, on average, the proposed dynamic pipeline-partitioning video decoder is 34% faster compared to a wavefront-based parallel video decoder.

In this paper, we present the design of a four-core Java application processor, JAIP-MP. Each processor core in JAIP-MP is a hardwired Java core that supports dynamic class loading, two-fold bytecode execution, object-oriented dynamic resolution, method and object caching, Java exception handling, and temporal multithreading. For JAIP-MP, a global load-balancing task manager is used to evenly distribute Java threads among the local task queues of every processor cores. In addition, a data coherence controller is designed to enforce coherence across all data caches and to perform synchronization operations among Java threads of all processor cores. Since thread management and synchronization mechanisms are completely implemented in hardware, the single-core multi-tasking performance of JAIP-MP is much higher than that of a software-based VM running on a traditional OS kernel such as Linux. For execution of multithreading applications, the speedup of a four-core JAIP-MP system can be up to 3.69 times faster than a single-core JAIP system, tested using the JemBench parallel benchmark programs.

In this paper, we propose a new multicore application processor architecture that facilitates the adoption of the fine-granularity software-pipeline parallelism without causing an extra burden on the system bus. The proposed system-on-a-chip architecture can simultaneously support the traditional symmetric multiprocessor (SMP) and the proposed software-pipeline applications efficiently. The programming model of the proposed architecture is compatible with the existing SMP operating systems. For the implementation of the pipeline-based parallelism, new programmer-friendly system calls are suggested to take advantage of the new software-pipeline datapath. The proposed architecture with four reduced instruction set computing cores is implemented on an field-programmable gate array development board for verification. An Advanced Video Coding/H.264 baseline profile video decoder that explores the pipeline parallelism with dynamic pipeline-stage partitioning is implemented on the target platform to justify the benefits of the proposed architecture. Experimental results show that the adoption of the proposed pipeline datapath architecture into existing application processors enables new potentials in exploring software parallelism.

In this article, we present a reusable Java processor IP for application processors of embedded systems. For the Java microarchitecture, we propose a low-cost stack memory design that supports a two-fold instruction folding pipeline and a low-complexity Java exception handling hardware. We also propose a mapping between the Java dynamic class loading model and the SoC platform-based design principle so that the Java core can be encapsulated as a reusable IP. To achieve this goal, a two-level method area with two on-chip circular buffers is proposed as an interface between the RISC core and the Java core. The proposed architecture is implemented on a Xilinx Virtex-5 FPGA device. Experimental results show that its performance has some advantages over other Java processors and a Java VM with JIT acceleration on a PowerPC platform.

The performance and characteristics of the storage devices used in embedded systems can have a great influence on the overall end user experience. When building embedded systems or designing new storage device components, it is important for the designers to be able to evaluate how storage devices of different characteristics will affect the overall system performance. Storage device emulation enables a system's performance to be evaluated with simulated storage devices that are not yet available. In storage device emulation, the emulated storage device appears to the operating system (OS) as a real storage device and its service timings are determined by a disk model, which simulates the behavior of the target storage device. In the conventional storage device emulators, because the OS is running continuously in the real-time domain, the amount of time that the emulators can spend on processing each I/O request is limited by the service time of each corresponding I/O request. This timing constraint can make emulating high-speed storage devices a challenge for the conventional storage device emulators. In this article, we propose an OS state pausing approach to storage device emulation that can overcome the timing constraints faced by the conventional storage device emulators. By pausing the state of the OS while the storage device emulator is busy, the proposed emulator can spend as much time as it needs for processing each I/O request without affecting the performance of the emulated storage device as perceived by the OS. This allows the proposed storage device emulator to emulate storage devices that would otherwise be challenging or even impossible for the conventional storage device emulators. In addition, the main task of storage device emulation is offloaded to an external computer to minimize the impact of the emulation workload on the target machine. The proposed storage device emulator is implemented with the Linux OS 1 on an embedded system development board. Experimental results show that the full-system performance benchmarks measured with the proposed storage device emulator are within 2% differences compared to the results of the reference system.

In this paper, we presents the design of a hardware temporal multi-threading architecture for a Java processor. The Java virtual machine (JVM) model is a stack machine where the process state is the snapshot of the Java stack. If the runtime stack is stored (or cached) in on-chip memory for performance reasons, the backup and restoration of the Java runtime stacks for context switching would be expensive operations. We propose a Ping-Pong buffer architecture in this paper to facilitate fully hardware-based multi-threading capability for a Java processor. The proposed hardware architecture has been implemented and verified on an FPGA platform, Xilinx ML605. The experimental results show that the proposed context-switching efficiency is much higher than that of a software-based VM such as the CVM-JIT. Therefore, the proposed hardwired Java processor is promising for embedded applications that require heavy multi-threading operations.

In this paper, a new motion compensation scheme based on dense true motion fields is investigated for video coding. The proposed technique is similar to the block-based decoder-side motion estimation method except that dense motion fields estimated from previously decoded frames are used for frame prediction. This paper demonstrates that even with compression distortions, an iterative dense true motion estimation technique can still produce good quality dense motion fields for video coding or frame-rate up-sampling applications. In particular, we have shown that when the new dense motion field compensation tool is integrated into the B-frame codec of H.264, one can achieve up to 8% overall BD bit-rate savings using standard MPEG test sequences.

This article presents the design of a video decoder using dynamic task partition approach on a heterogeneous dual-core embedded platform. For such systems, static task partition between the two cores at design time is a typical approach for application development. In this article, we propose a runtime dynamic task partition model and implement an MPEG-4 Simple Profile video decoder using this approach on a TI OMAP 5912 platform. Comparing with a traditional mobile video decoder optimized for the same DSP core, the performance gain from dynamic task partition is 38.4% on average. More importantly, the gain is achieved with the design constraint that the implementation effort for the dynamic task partition decoder is about the same as the effort using design-time task partition model. Unlike common belief that the inter-processor communication overhead would be too high to justify intense cooperation between two heterogeneous cores, this paper shows that it is indeed beneficial to adopt dynamic task partition model on commercially available heterogeneous multi-core platforms.

In this paper, we propose the design of the stack memory for a low-cost Java processor that explores instruction-level parallelism. The Java virtual machine (JVM) is a stack machine where the instruction execution pipeline uses a stack to store intermediate computation results and local variables. High performance Java processors often use a large stack cache to enable parallel accesses to operands and local variables to achieve instruction-level parallelism. We propose a low-cost alternative of stack memory design that allows the Java processor to access the critical stack operands and local variables concurrently. The stack memory is constructed using seven registers and two blocks of dual-port on-chip SRAM; and is optimized for the Java instruction set architecture. When coupled with a low-cost two-way instruction folding pipeline, micro-benchmark results show that the proposed architecture can achieve up to 45.4% 2-fold instruction folding rate.

In this paper, a dense motion field estimation technique based on the Bayesian framework is proposed to estimate the true dense motion fields of video sequences. Previous stochastic techniques of dense motion field estimation adopts piecewise smooth motion model and use MAP estimation to find the motion field with joint minimization of motion compensation errors and maximization of motion smoothness. However, such random process does not guarantee to converge to the true motion field. In this paper, the motion of landmark points in the video sequence is introduced into the MAP estimation process to regularize the estimated motion field. Experimental results show that the proposed algorithm produces estimated motion fields which preserve piecewise smooth nature and are visually close to the true motion of the video sequences.

In this paper, we have presented a low-cost Java class caching mechanism for Java processors. The design is integrated into a heterogeneous dual-core Java SoC that is targeted for embedded multimedia applications with GUI support. The design goal of the proposed caching mechanism is to improve the Java program execution performance without the complexity of a multi-way set associative cache. Since object-oriented Java programs are usually composed of many small classes, the proposed low-cost caching mechanism works fairly well for practical applications. The Java SoC is implemented and emulated on an Xilinx Virtex-4 device and the benchmark results show that the performance of the proposed dual-core Java SoC is faster than other popular software-based VMs for embedded systems.

In this paper, we have proposed a fast inter-processor communication interface (IPC) for a dual-core Java application processor. The dual-core Java application processor is a SoC composed of a RISC core and a double-issued Java bytecode execution core. The proposed fast IPC mechanism provides Java system software a high-level way to invoke a host processor service routine from Java source code. The proposed IPC has much lower overhead than that of the standard Java Native Interface (JNI). Unlike other fast native call interface designed for VM interpreter, the proposed IPC mechanism is exclusively designed for the communication between two physical hardwired processor cores. Based on the experimental results, the proposed mechanism is very promising for embedded Java runtime environment.

In this paper, we have proposed an efficient hardware-assisted syntax decoding model for software-based video decoder. The proposed syntax decoding model is a generic model for different video codec standards. The syntax decoding process is divided into codec-dependent high-level syntax parser and generic entropy decoding engines. Currently, the design is implemented specifically for the support of AVC/H.264 standard (for both CAVLC and CABAC acceleration). Nevertheless, the design of the proposed syntax decoding model has the potential of becoming the design of a flexible bitstream parser, which is the most challenging problem in the MPEG Reconfigurable Video Coding (RVC) Framework. A Virtex-5 FPGA development board is used to implement and verify the full hardware-software system (including the hardware entropy engines and the software syntax parser and macroblock data reconstruction modules extracted from JM12.2).

In this paper, a distributed video coding technique with prioritized channel coding of W-Z frames is presented. In the proposed framework, W-Z frame macroblocks are classified into different groups based on the estimated quality of the side information. The information is transmitted via uplink channel back to the encoder so that macroblocks with similar error statistics can be grouped tighter in same coding blocks for channel coding. With this approach, decoder can request more parity bits to correct macroblocks whose side information quality is worse and request less parity bits for macroblocks with smaller side information errors. Initial experimental results show that the proposed technique can improve the R-D performance of both pixel-domain and transform-domain DVC codecs considerably.

Due to the variety of popular video coding standards, many efforts have been put into the design of a single video decoder chip that supports multiple formats. In 2004, ISO/IEC MPEG started a new work item to facilitate multi-format video codec design and to enable more flexible usage of coding tools. The work item has turned into the MPEG reconfigurable video coding (RVC) framework. The key concept of the RVC framework is to allow flexible reconfiguration of coding tools to create different codec solutions on-the-fly. In this paper, flexible SoC architecture is proposed to support the RVC framework. Some analysis has been conducted to show the extra costs required for this platform compared to hard-wired codec architecture. In conclusion, the RVC framework can be mapped to an SoC platform to provide flexibility and scalability for dynamic application environment with reasonable cost in hardware design.

Java applications for embedded systems are becoming popular today. CLDC/MIDP is the standard application platform for mobile phones while CDC/PBP is the emerging application platform for next generation digital TV set-top boxes. Although software-based Java virtual machines (VM) are prevalent, most of these VMs require a host processor running at much higher clock rate than 300MHz to reach reasonable performance. This is beyond the recommended specification of handsets and set-top boxes. In this paper, we have proposed a double-issue Java processor for embedded systems. The design is not tied to any host processors and can be used as an efficient binary execution engine for a full Java runtime environment implementation. When synthesized on a Virtex IV FPGA (4VFX12FF66-10), the RTL model can reach over 100MHz and consumes less than 22% resources of the device.

A video bit allocation technique adopting a visual distortion sensitivity model for better rate-visual distortion coding control is proposed in this paper. Instead of applying complicated semantic understanding, the proposed automatic distortion sensitivity analysis process analyzes both the motion and the texture structures in the video sequences in order to achieve better bit allocation for rate-constrained video coding. The proposed technique evaluates the perceptual distortion sensitivity on a macroblock basis, and allocates fewer bits to regions permitting large perceptual distortions for rate reduction. The proposed algorithm can be incorporated into existing video coding rate control schemes to achieve same visual quality at reduced bitrate. Experiments based on H.264 JM7.6 show that this technique achieves bit-rate saving of up to 40.61%. However, the conducted subjective viewing experiments show that there is no perceptual quality degradation. EDICS-1-CPRS, 3-QUAL.

A novel video bit allocation technique adopting a visual distortion sensitivity model for better rate-visual distortion coding control is proposed in this paper. Instead of applying complicated semantic understanding, the proposed automatic distortion sensitivity analysis process analyzes both the motion and the texture structures in the video sequences in order to achieve better bit allocation for rate-constrained video coding. This analysis evaluates the tolerable perceptual distortions on a macroblock basis, and allocates fewer bits to regions permitting large perceptual distortions for rate reduction. The proposed algorithm can be incorporated into any existing video coding rate control schemes to achieve same visual quality at greatly reduced bitrate. Experiments based on H.264 show that this technique achieves bit-rate saving of up to 40% with no perceptual quality degradations. The experiments also demonstrate the inadequacy of using PSNR as a distortion measure in a video coding framework.

The paper proposes a novel video coding approach that performs adaptive rate-distortion optimization guided by perceptual hints. The key idea is to adjust adaptively the Lagrange multipliers of the RDO coder control module based on visual attention analysis. The observation is that human vision is sensitive to movement of well-structured objects while tolerating large distortion in moving areas with random structure (texture-wise and motion-wise). The proposed algorithm analyzes permissible perceptual distortions and, accordingly, assigns larger Lagrange multipliers to regions that are perceptually less sensitive to distortion so that rate reduction is weighted more than distortion reduction in these regions. Experiments show that this scheme achieves bit-rate saving of 5-7% with virtually the same perceptual quality and is very promising for practical systems.

Binocular camera systems are commonly used to construct 3-D-based scene description. However, there is a tradeoff between the length of the camera baseline and the difficulty of the matching problem and the extent of the field of view of the 3-D scene. A large baseline system provides better depth resolution than a smaller baseline system at the expense of a narrower field of view. To increase the depth resolution without increasing the difficulty of the matching problem and decreasing the field of view of the 3-D scene, a sequential 3-D-based scene description technique is proposed. Multiple small-baseline 3-D scene descriptions from a single moving camera or an array of cameras are used to sequentially construct a large baseline 3-D scene description while maintaining the field of view of a small-baseline system. A Bayesian framework using a disparity-space image (DSI) technique for disparity estimation is presented. The cost function for large baseline image matching is designed based not only on the photometric matching error, the smoothness constraint, and the ordering constraint, but also on the previous disparity estimates from smaller baseline stereo image pairs as a prior model. Texture information is registered along the scan path of the camera(s). Experimental results demonstrate the effectiveness of this technique in visual communication applications.

Many optical flow estimation techniques are based on the differential optical flow equation. These algorithms involve solving over-determined systems of optical flow equations. Least squares (LS) estimation is usually used to solve these systems even though the underlying noise does not conform to the model implied by LS estimation. To ameliorate this problem, work has been done using the total least squares (TLS) method instead. However, the noise model presumed by TLS is again different from the noise present in the system of optical flow equations. A proper way to solve the system of optical flow equations is the constrained total least squares (CTLS) technique. The derivation and analysis of the CTLS technique for optical flow estimation is presented in this paper. It is shown that CTLS outperforms TLS and LS optical flow estimation.

A technique for constructing a sequential 3D-based scene description is proposed in this paper. Multiple small baseline 3D scene descriptions from a single moving camera or an array of cameras are used to sequentially construct a large baseline 3D scene description while maintaining the field of view of a small baseline system. A Bayesian framework using a disparity-space image (DSI) technique for disparity estimation is presented. The cost function for large baseline image matching is designed based not only on the photometric matching error, the smoothness constraint, and the ordering constraint, but also on the previous disparity estimates from smaller baseline stereo image pairs as a prior model. Texture information is registered along the scan path of the camera(s). Experimental results demonstrate the effectiveness of this technique in visual communication applications.

The problem of the enhancement of a low bit-rate compressed video sequence using the information provided by the encoder is investigated in this paper. The proposed algorithm is spatio-temporally adaptive and enforces different degrees of between-block, within-block, and temporal smoothness of the decompressed frames based on macroblock types. The algorithm uses the projections onto the sets that capture the information conveyed by the transmitted data to constrain the solution space. A partially automatic regularization parameter estimation algorithm is also proposed in this paper. An analysis of PSNR gain per macroblock type is presented to cast insight into the compressed video enhancement problem. The experimental results demonstrate the effectiveness of the proposed approach.

A new divide-and-conquer technique for disparity estimation is proposed in this paper. This technique performs feature matching following the high confidence first principle, starting with the strongest feature point in the stereo pair of scanlines. Once the first matching pair is established, the ordering constraint in disparity estimation allows the original intra-scanline matching problem to be divided into two smaller subproblems. Each subproblem can then be solved recursively until there is no reliable feature point within the subintervals. This technique is very efficient for dense disparity map estimation for stereo images with rich features. For general scenes, this technique can be paired up with the disparity-space image (DSI) technique to compute dense disparity maps with integrated occlusion detection. In this approach, the divide-and-conquer part of the algorithm handles the matching of stronger features and the DSI-based technique handles the matching of pixels in between feature points and the detection of occlusions. An extension to the standard disparity-space technique is also presented to compliment the divide-and-conquer algorithm. Experiments demonstrate the effectiveness of the proposed divide-and-conquer DSI algorithm.

We propose a new method for disparity assisted stereo optical flow estimation. This method is based on the linearization of the round-about compatibility constraint, which converts a stereo optical flow estimation problem to a single channel optical flow estimation problem. An over-determined system of optical flow equations can then be constructed for estimating the flow fields. The total least squares, instead of the traditional least squares is used to solve the estimation problem. We also investigate the extension of the locally constant flow model across the time domain. The experiments presented demonstrate that the proposed technique performs very well.

In this paper a model-based multi-view image generation system for video conferencing is presented. The system assumes that a 3-D model of the person in front of the camera is available. It extracts texture from speaking person sequence images and maps it to the static 3-D model during the videoconference session. Since only the incrementally updated texture information is transmitted during the whole session, the bandwidth requirement is very small. Based on the experimental results one can conclude that the proposed system is very promising for practical applications.

Two statistical approaches for 3-D reconstruction from an image sequence are compared: the asymptotic Bayesian surface reconstruction and the Kalman filter-based depth estimation. Both techniques are recursive algorithms where relevant information contained in previously taken images is summarized in a prior term (prior to the taking of the next image). This means that the reconstruction results are based upon information from all images but the storage and computation required do not grow dramatically. Experiments with both real images and computer generated images demonstrate that the asymptotic Bayesian approach achieves better results than the Kalman filter-based approach, largely due to better problem formulation.>