To model the periodicity of beats, state-of-the-art beat tracking systems use “post-processing trackers” (PPTs) that rely on several empirically determined global assumptions for tempo transition, which work well for music with a steady tempo. For expressive classical music, however, these assumptions can be too rigid. With two large datasets of Western classical piano music, namely the Aligned Scores and Performances (ASAP) dataset and a dataset of Chopin's Mazurkas (Maz-5), we report on experiments showing the failure of existing PPTs to cope with local tempo changes, thus calling for new methods. In this paper, we propose a new local periodicity-based PPT, called predominant local pulse-based dynamic programming (PLPDP) tracking, that allows for more flexible tempo transitions. Specifically, the new PPT incorporates a method called “predominant local pulses” (PLP) in combination with a dynamic programming (DP) component to jointly consider the locally detected periodicity and beat activation strength at each time instant. Accordingly, PLPDP accounts for the local periodicity, rather than relying on a global tempo assumption. Compared to existing PPTs, PLPDP particularly enhances the recall values at the cost of a lower precision, resulting in an overall improvement of F1-score for beat tracking in ASAP (from 0.473 to 0.493) and Maz-5 (from 0.595 to 0.838).

For expressive music, the tempo may change over time, posing challenges to tracking the beats by an automatic model. The model may first tap to the correct tempo, but then may fail to adapt to a tempo change, or switch between several incorrect but perceptually plausible ones (e.g., half- or double-tempo). Existing evaluation metrics for beat tracking do not reflect such behaviors, as they typically assume a fixed relationship between the reference beats and estimated beats. In this letter, we propose a new performance analysis method, called annotation coverage ratio (ACR), that accounts for a variety of possible metric-level switching behaviors of beat trackers. The idea is to derive sequences of modified reference beats of all metrical levels for every two consecutive reference beats, and compare every sequence of modified reference beats to the subsequences of estimated beats. We show via experiments on three datasets of different genres the usefulness of ACR when being utilized alongside existing metrics, and discuss the new insights that can be gained.

This letter presents a novel system architecture that integrates blind source separation with joint beat and downbeat tracking in musical audio signals. The source separation module segregates the percussive and non-percussive components of the input signal, over which beat and downbeat tracking are performed separately and then the results are aggregated with a learnable fusion mechanism. This way, the system can adaptively determine how much the tracking result for an input signal should depend on the input's percussive or non-percussive components. Evaluation on four testing sets that feature different levels of presence of drum sounds shows that the new architecture consistently outperforms the widely-adopted baseline architecture that does not employ source separation.

Blind music source separation has been a popular and active subject of research in both the music information retrieval and signal processing communities. To counter the lack of available multi-track data for supervised model training, a data augmentation method that creates artificial mixtures by combining tracks from different songs has been shown useful in recent works. Following this light, we examine further in this paper extended data augmentation methods that consider more sophisticated mixing settings employed in the modern music production routine, the relationship between the tracks to be combined, and factors of silence. As a case study, we consider the separation of violin and piano tracks in a violin piano ensemble, evaluating the performance in terms of common metrics, namely SDR, SIR, and SAR. In addition to examining the effectiveness of these new data augmentation methods, we also study the influence of the amount of training data. Our evaluation shows that the proposed mixing-specific data augmentation methods can help improve the performance of a deep learning-based model for source separation, especially in the case of small training data.

For the patients suffering from amyotrophic lateral sclerosis (ALS), they are encouraged to exercise their bodies routinely to prevent or delay the paralysis of muscles. This work proposes an electromyogram (EMG) based ALS rehabilitation system via playing computer games. The multi-channel EMG measuring system and the controlled interfaces to computer games were developed. According to the symptoms of disability, the controls from different muscles are designed. For ALS patients in the early stage, the EMG electrodes were placed on the forearm to detect the finger gestures; for ALS patients in the middle stage, the EMG signals of upper extremity were employed to detect the hand gestures and arm moving; for the late ALS stage, the EMG electrodes were placed on chin to detect the facial expression. A commercial video game as well as a self-modified computer game are utilized in our rehabilitation systems. We believe that the patients are more preferable to exercise their bodies in a form of entertainment.

This study presents a discussion on the task of score alignment, which properly aligns an audio recording with its corresponding score. Conventional methods have difficulty performing this task because of asynchrony in the recording of simultaneous notes in the score. A note‐based score alignment based on the pitch‐by‐time feature is proposed, called the piano‐roll feature, and it presents an approach for converting the audio spectrogram to a piano‐roll‐like feature. Score‐driven non‐negative matrix factorisation is then adopted in the transformation. Furthermore, this study also proposes pitch‐wise alignment considering each pitch sequence (i.e. the row of piano roll) separately. Results based on the MIDI‐Aligned Piano Sounds database show that approximately 88% of notes match their onsets, deviating from the ground truth by less than 50 ms. Other results based on SCREAM Music Annotation Project database that is a manual annotation project of commercial CD recordings are presented as well.

Scratchpad memory (SPM) has been increasingly used in embedded systems due to its higher efficiency in terms of energy and area compared to that of ordinary cache. A hybrid on-chip memory architecture that combines SPM with a mini-cache has been proposed. One key issue for hybrid on-chip memory architectures is to reduce the number of off-chip memory accesses and energy consumption. Existing methods achieve this by moving the most frequently accessed data into SPM. However, these methods may be ineffective because the main source of off-chip memory accesses may not be the most frequently accessed data. Instead, most off-chip memory accesses are caused by cache misses, so reducing the latter will reduce the former. Cache misses are mainly caused by data contending for cache lines. Therefore, this paper proposes a contention-aware SPM allocation method for hybrid on-chip management. The number of cache misses for a page is used as a metric to determine whether a page should be moved to SPM. When the number of misses for a page exceeds a threshold, the page is moved to SPM, reducing cache contention. Experimental results show that the proposed method can reduce the energy delay product by 35% to 53% compared to a cache-only on-chip memory architecture and 19% to 31% compared to an existing hybrid on-chip memory architecture.

The coding efficiency of a still image coder is dominated by the complexity of the encoding algorithm. For the past decade, it was seen that the encoding quality was gradually limited. Hence, the issue of coding speed becomes popular in recent researches. This paper proposes a block-based image coder which is applicable for parallel computing. The proposed coder improves the coding quality of combined significant map coding (CSMC), and shortens the encoding time dramatically. From the simulation results, the proposed method has comparable coding quality against the most well-known coders. The coding quality is not affected by the number of participants, so it is also suitable to be applied in cloud computing. Besides, the proposed block-based image coder preserves the property of high scalability and availability. We conclude that our proposed coder enabled by parallel computing has a great advantage of fitting with the increasing complexity of computation.

Hough transform is widely used for detecting straight lines in an image, but it involves huge computations. For embedded application, field-programmable gate arrays are one of the most used hardware accelerators to achieve real-time implementation of Hough transform. In this paper, we present a resource-efficient architecture and implementation of Hough transform on an FPGA. The incrementing property of Hough transform is described and used to reduce the resource requirement. In order to facilitate parallelism, we divide the image into blocks and apply the incrementing property to pixels within a block and between blocks. Moreover, the locality of Hough transform is analyzed to reduce the memory access. The proposed architecture is implement on an Altera EP2S180F1508C3 device and can operate at a maximum frequency of 200 MHz. It could compute the Hough transform of 512 × 512 test images with 180 orientations in 2.07-3.16 ms without using many FPGA resources (i.e., one could achieve the performance by adopting a low-cost low-end FPGA).

In this paper, we use an FPGA platform and a high level synthesis tool, called Impulse C, to speedup a statistical Line Of Reaction (LOR) estimation for a high-resolution Positron Emission Tomography (PET) scanner. The estimation algorithm provides a significant improvement over conventional methods, but the execution time is too long to be practical for clinic applications. Impulse C allows us to rapidly map a C program into a platform with a host processor coupled to an FPGA device. However, the generated HDLs from the original codes are very inefficient, and the execution time is even worse than the software code. We describe some optimization methods for the algorithm using Impulse C. These methods could also be applied to other applications or used to improve the high level synthesis tools. The results show that the FPGA implementation can obtain a 82× speedup over the optimized software.

Fingerprints have been used to accurately identify people since the nineteenth century. However, as more and more persons included into the repository, the size of database grows explosively. Storing the high-quality fingerprint images in low bit-rate thus becomes necessary. In this paper, a novel DCT-based coder is developed for fingerprint compression by using the specific energy distributions of fingerprint patterns. An adaptive scheme, which utilizes spatial-oriented tree (SOT) construction on the transformed image blocks, is proposed to improve the compression process. Consequently, the proposed coding scheme yields the enhancement of 0.5 to 2.5 dB over WSQ, SPIHT and JPEG2000 for fingerprint patterns at the same compression ratio. The computation complexity of this method is O(n).

In SOT based coding methods, ordering of the coefficients could affect coding efficiency. A small coefficient placed close to tree root usually reduces the coding efficiency significantly. Hence, a good reordering scheme for a SOT is critical. In digital audio coding, information of neighboring frames is usually highly correlated. Based on this property, a SOT based audio coding scheme with a coefficient reordering method is proposed. The proposed coder may have the coefficients of the current frame reordered depending on the decoded results of some previous frames. The coding qualities of the proposed coder and MPEG-4 AAC coder are compared using ODG (Objective Differential Grade) on a variety of sound sources.

Multicore processor provides large computation capability but also involves the complicate parallel programming. One of major considerations in parallel programming is the performance. Traditional design methodologies which usually start a design on a selected platform spend a lot of effort and time on tuning performance and debugging. When platform is changed even with different number of cores, considerable redesign effort is required. Hence a flexible design methodology is necessary. In this paper, a design methodology is presented for video codec, by using MPEG-4 SP decoder as an example, on multicore processor. The parallelisms of MPEG-4 decoder are discussed and exposed with the dataflow model. The dataflow model provides a high-level abstraction of underlying hardware. Computation and communication of MPEG-4 decoder are separated and represented as modules and channels, respectively. It is possible to synthesize the model targeting to either dedicate hardware or software on multiprocessor. To map the high level dataflow model to Cell processor, the mapping flow, including offline profiling, task allocation and runtime libraries, are developed. According to the profiling results, the allocation algorithm could allocate tasks on multiprocessors as balanced as possible. An efficient synchronization mechanism on Cell processor is also proposed. We also discuss the impact of the model and the mapping flow corresponding to decoding speed. The results show that the proposed methodology gets considerable performance boost when the number of cores is increased.

Previous researches show that a scratchpad memory device consumed less energy than a cache device with the same capacity. In this article, we locate the scratchpad memory (SPM) in the top level of the memory hierarchy to reduce the energy consumption. To take the advantage of a SPM, we address two issues of utilizing a SPM. First, the program's locality should be improved. The second issue is SPM management. To tackle these two issues, we present a hardware/software framework for dynamically allocating both instructions and data in SPM. The software flow could be divided into three phases: locality improving, locality extraction, and runtime SPM management. Without modifying the original compiler and the source code, we improve the locality of a program. An optimization algorithm is proposed to extract the SPM allocations. At runtime, an SPM management program is employed. In hardware, an address translation logic (ATL) is proposed to reduce the overhead of SPM management. The results show that the proposed framework can reduce energy delay product (EDP) by 63%, on average, when compared with the traditional cache architecture. The reduction in EDP is contributed by properly allocating both instructions and data in SPM. By allocating only instructions in SPM, the EDPs are reduced by 45%, on average. By allocating only data in SPM, the EDPs are reduced by 14%, on average.

Evaluating coding efficiency in early coding stage is important in modern RDO compression methods. Bit-plane coding algorithms such as SPIHT are widely used in many scalable coders. The locality of a spatial-oriented tree (SOT) affects the coding efficiency of a SPIHT coder. In this paper, a fast algorithm is proposed to determine the numbers of bits to be consumed at all coding levels for a SPIHT coder without performing the entire coding process. The proposed algorithm can also evaluate the localities of various SOTs so that one can decide which types of SOTs should be used. Experiments of using different coefficient ordering schemes on construct SOTs for image compression using SPIHT are presented.

We present an effective congestion and error control mechanism for scalable video coding (SVC) extension of the H.264/AVC video dissemination over Internet. The congestion control is used to determine the appropriate number of SVC video layers based on bandwidth inference congestion (BIC) control protocol for layered multicast scenarios and the error control is achieved by unequal forward error correction (FEC) layered protection using block erasure coding. Through the real Internet streaming experiments, we demonstrate the effectiveness of the proposed layered SVC delivery, in terms of subscription layer, average packet loss rate and PSNRs, under several layered-definition scalabilities.

In this paper a two-stage mode selection (TSMS) algorithm is presented to speed up the H.264/AVC video encoding process with rate distortion optimization (RDO). However, lots of additional computing power is required and this makes the realization of H.264/RDO in a resource-limited system very difficult. The proposed TSMS employs a two-stage decision process: the first stage is to predict some probable encoding modes according to the information when one encodes the preceding macroblocks and video frames. The second stage refines the decision with techniques based on Baye's probability rule and Back-Propagation neural network (BPN). According to the experiment results, over 50% of the computation time is reduced with very slight loss in peak signal-to-noise ratio (PSNR) and a slightly increment in bit rate when TSMS is applied. The TSMS is even faster than the encoding program with running RDO part. All programs are based on the H.264/AVC standard reference software (JM 9.2).

Struck-string instruments such as pianos usually have groups of strings that terminate at some common bridges. Because of the strong coupling phenomenon, the produced tones can exhibit highly complex modulation patterns, and synthesizing such complex tones turns out to be quite complicated. It is also difficult to determine the synthesis model parameters such that the synthesized tones can match the recorded tones. This paper proposes a multichannel recurrent network based on three previous works: the coupled-string model, the commuted piano synthesis method, and the infinite impulse response (IIR) synthesis method. This work attempts to automatically extract the synthesis parameters by using a neural-network training algorithm without the knowledge of the instruments' physical properties. Encouraging results are shown in the computer simulations

MPEG-4 advanced simple profile video provides I, P, and B-type frames in each GOP (group of pictures). To maximize the coding efficiency, it is important to determine the distribution of the three frame types, also called the GOP structure. In this paper, the GOP structure is determined dynamically in real time by using the information, such as Sad (sum of absolute difference) and Mad (mean of absolute difference), generated in the encoding process and little amount of computation is used. The algorithm first determines whether the current frame is an intra-picture or an inter-picture. If it is an inter-picture, one has to decide whether it is a P_picture or a B_picture. No more than 4 consecutive B_pictures can be employed to reduce the video buffer size. The proposed method is tested over a wide range of video sequences at different data rate conditions and produces consistent improvement compared to fixed GOP methods

A novel neural network based method for halftoning and inverse halftoning of digital images is presented. We first start from inverse half-toning of images produced from error diffusion methods using an RBF network plus an MLP network. The restored contone images have had good quality already. Then, an SLP neural network is used to refine the halftoning processing and the training process of the inverse half-toning network is also involved. The combined training procedure produces half-tone images and the corresponding continuous tone images at the same time. It is found that these contone images have even better PSNR performance. Furthermore, the resulted half-tone images are visually sharper and clearer, too. The proposed inverse half-toning method is also compared to the well-known LUT method.

Wavelet based compression approaches becomes very popular in the last few years. In this paper, a method called CEIHT (concurrent encoding in hierarchical trees) rooted from SPIHT is proposed. This method tries to explore the relationship between adjacent resolution levels and encodes multiple coefficients with three fixed Huffman tables at the same time. Consistent coding efficiency improvement is achieved compared to the original SPIHT. Though wavelet based compression methods usually encode on a whole image, block based approaches have to be applied in many low-cost embedded applications because their memory space and computation power are usually limited. The proposed method is especially efficient in such cases. The performance of the proposed method is compared favorably to the highly acclaimed EBCOT based J2K when small coding block sizes are necessary.

An error concealment using non-uniform rational B-spline (NURBS) is proposed. NURBS has been employed by many CAD/CAM systems as a fundamental geometry representation. Despite the fact that NURBS has gained tremendous popularity from the CAD/CAM and computer graphics community, its application on exploring the image problem only received little attention. On the other hand, the image contents might be corrupted or lost during transmission. Although there are quite a few existing techniques, yet developing an effective approach to conceal the error remains as one of the hottest research topics. Thus the aim of this study is to develop an image reconstruction technique using NURBS. The key idea is to use NURBS to represent the portion of image data without corruption. By accomplishing this, a single-hidden layer neural network is employed to learn the appropriate control points of NURBS. After learning, NURBS is then used to render the corrupted image data. Experimental results indicate that the proposed approach exhibits promising performance.

A new approach is proposed that closely synthesizes tones of plucked string instruments by using a class of physical modeling recurrent networks. The strategies employed consist of a fast training algorithm and a multistage training procedure that are able to obtain the synthesis parameters for a specific instrument automatically. The training vector can be recorded tones of most target plucked instruments with ordinary microphones. The proposed approach delivers encouraging results when it is applied to different types of plucked string instruments such as steel-string guitar, nylon-string guitar, harp, Chin, Yueh-chin, and Pipa. The synthesized tones sound very close to the originals produced by their acoustic counterparts. In addition, the paper presents an embedded technique that can produce special effects such as vibrato and portamento that are vital to the playing of plucked-string instruments. The computation required in the resynthesis processing is also reasonable.

In this paper, an automatic way of analyzing/synthesizing plucked-string instruments by using a simple IIR (infinite impulse response) structure is presented. The basic structure of the proposed IIR filter consists of a delay line with feedback loops pulled out from the delay line back to the input side of the filter. The feedback portion acts like a prediction filter to modify the signal for the next period such that it can be as close to the original tone as possible. The delay line and the prediction filter provide the mechanism to produce the desired pitch of the synthetic sound. The design of the coefficients for this IIR synthesizer is accomplished by using a neural network based training algorithm. The excitation signal is extracted from the original tones recorded from the target instrument. Several sets of filter coefficients are required for different synthesis stages such as the attacking part and sustain part. The proposed algorithm is tested over a wide range of plucked-string instruments. Although it requires more computation compared to some model-based approaches, the analysis work is simple and straightforward. In addition, the synthetic tone sounds very close to the correspondent original sound.

A physical modeling method for electronic music synthesis of plucked-string tones by using recurrent networks is proposed. A scattering recurrent network (SRN) which is used to analyze string dynamics is built based on the physics of acoustic strings. The measured vibration of a plucked string is employed as the training data for the supervised learning of the SRN. After the network is well trained, it can be regarded as the virtual model for the measured string and used to generate tones which can be very close to those generated by its acoustic counterpart. The "virtual string" corresponding to the SRN can respond to different "plucks" just like a real string, which is impossible using traditional synthesis techniques such as frequency modulation and wavetable. The simulation of modeling a cello "A"-string demonstrates some encouraging results of the new music synthesis technique. Some aspects of modeling and synthesis procedures are also discussed.

A new recurrent-network-based synthesis technique for general plucked-string instruments is proposed. If is expected that the proposed technique can serve as a synthesis model as well as a systematic parameter-searching engine such that the synthetic tones can sound as close to the recorded tone as possible. Two traditional Chinese plucked-string instruments, Pipa and Qin, are introduced. A multi-stage training procedure and an improved training algorithm for the determination of synthesis parameters and excitation signal are presented. The analysis/synthesis results of the two Chinese traditional plucked-string instruments considered are shown.

Physical modeling music synthesis techniques by simulating the vibrations of acoustic instruments are becoming more and more popular in the recent years. In the past few years, trying to find those appropriate parameters in order to make the synthesis result sound close to a specific instrument has been a very difficult problem. Various try-and-error methods based on frequency domain and time domain approaches were used mostly. In this paper, we propose a systematic analysis/synthesis model called the Scattering Recurrent Network (SRN) which is capable of modeling the dynamics of a plucked string and synthesizing tones which are very close to the tones produced by the same string. The sound quality is superior to both FM and Wavetable synthesis techniques.