A compact implementation of a foreground segmentation processor in a multi-resolution transform domain has been proposed for HDTV signals. The proposed architecture is designed to simplify system controls by the hardware streaming and to reduce required memory capacities. It enables flowing pixels through all functional units in order, including multi-resolution spatial transform and temporal segmentation. The resultant architecture does not use memories except I/O buffers. Therefore, memory modules as well as complex address manipulation over the multiple global transforms and spatial/temporal interface is not required. The FPGA prototype chip dissipates 150 mW of power. This approach can be used for tablets and smart-phone by an ASIC implementation which will reduce the operation power to about 1/6.

This paper presents a no-reference (NR) based video-quality estimation method for compressed videos. The proposed method does not need bitstream information. Only pixel information is used for the quality estimation. An activity value which indicates a variance of luminance values is calculated for every given-size pixel block. The activity difference between an intra-coded frame and its adjacent frame is employed. In addition, blockiness and blur levels are also estimated at every frame and are taken into account. Experimental results show that the proposed method achieves accurate video-quality estimation. The correlation coefficient between subjective quality and estimated quality is 0.925. The proposed method is suitable for automatic quality check when the original videos cannot be used.

A simplified shadow removal approach by using interim results of transformed domain GMM foreground segmentation has been developed. The approach is based on the fact that the spatial frequency distribution does not change from the backgrounds in the shadow areas. Due to employing gray level picture processing and to utilizing only low frequency components in the transform domain, the resultant shadow removal approach drastically reduces the amount of processing, compared to conventional shadow removal approaches based on pixel based color component processing.

Simplified foreground Segmentation for a mobile video camera, such as a camera on a cell phone, is proposed. Unlike the static surveillance camera, a mobile camera is often used under various camera angles, including pan/tilt motions. The proposed algorithm, using a transformed domain Gaussian Mixture model, can realize segmentation under such variety of camera angles without any pre-stored background model. The camera parameters on a pan/tilt operation in a small angle change are considered to be translation and the translation amount is found out by POC (Phase only correlation) operations among a selected small number of blocks located at the transform block positions. The selection is based on criteria of excluding dynamic background blocks. The area which appears for the first time by a pan/tilt operation is processed by a boost background construction. Experimental results show the effectiveness of the approach.

This paper describes an efficient image enhancement method based on the multi-scale retinex (MSR) theory for applying to a motion picture pre-processing. The proposed method uses a multi-rate x-y separable Gaussian filter with a polyphase structure and conducts illumination domain MSR composition. Therefore, The processing amount is drastically reduced to 4-order less than that of original MSR. As enhancing dark and bright areas in an image is the main pre-processing purpose, the method modifies an enhancement function so as to improve color rendition of bright areas on an image. Although our approach aims at simple implementation, the proposed method shows similar performance to the original one and better performance than those of histogram equalization approaches.

This paper describes a precise and stable foreground segmentation using computationally efficient fine-to-coarse strategy based on a Gaussian mixture model (GMM). In our algorithm, a set of GMMs is employed on multiple block sizes by using Walsh transform (WT). Four neighboring WTs can be easily merged into a WT of four times wider block without using the inverse transform. The precise and stable processing comes from the multiresolutional GMM, and the WT spectral nature drastically reduces the computational steps. Experimental results show that our approach gives stable performance in many conditions, such as scenery in heavy snow and global lighting changes.

This paper describes an efficient architecture enhancement for video codec on a new-generation, general-purpose digital signal processor (DSP) core called SPXK5 developed for handheld devices. With high performance features of SPXK5's base architecture, an MPEG-4 video codec can be implemented efficiently. In addition, only a few SIMD type instructions effectively accelerate MPEG-4 video codec implementation by 20% with only 2.5% hardware increase. By reducing cycle count, the DSP's power consumption can be reduced. Both video and speech codec for 3G mobile service at 384kbps can be realized with a power consumption of less than 50mW.

Reed-Solomon (RS) coders are used for error-control coding in many applications such as digital audio, digital TV, software radio, CD players, and wireless and satellite communications. Traditionally, RS coders have been implemented using dedicated hardware. This paper considers software-based implementation of RS codecs. A hardware-software codesign approach is used to design the finite field datapath in a domain-specific digital signal processor (DSP) with low-energy RS codecs application in mind. These datapaths are designed to accommodate programmability with respect to the primitive polynomial as well as the field degree m. A novel heterogeneous digit-serial approach is proposed, where the heterogeneity corresponds to the use of different digit sizes in the multiply-accumulate (MAC) and degree reduction (DEGRED) subarrays. The salient feature of this digit-serial approach is that only the digit cells are implemented in hardware and the finite field multiplications are performed digit-serially in software by dynamically scheduling the internal digit-level operations. Efficient scheduling strategies for digit-serial finite field multiplications are presented and applied to the design of low-energy high-performance RS codecs in software. Significant energy and energy-latency reductions can be achieved using the digit-serial datapaths, as compared with the traditional approach where a combined MAC-DEGRED (parallel multiplier) unit is used. It is concluded that for two-error-correcting RS(n, k) codes over finite field GF(2/sup 8/), datapath containing a parallel MAC unit (of digit size eight) and a DEGRED unit with digit size two (or four) leads to RS codecs with the least energy consumption and energy-latency products; with these datapath architectures and appropriate digit-serial scheduling strategies, more than 60% energy reduction and more than one-third energy-latency reduction can be achieved compared with the parallel multiplication datapath-based approach.

Reed-Solomon (RS) codecs are used for error control coding in many applications such as digital audio, digital TV, software radio, CD players, and wireless and satellite communications. This paper considers software-based implementation of RS codecs where special instructions are assumed to be used to program finite field multiplication datapaths inside a domain-specific programmable digital-signal processor (DS-PDSP). A heterogeneous digit-serial approach is presented, where the heterogeneity corresponds to the use of different digit-sizes in the multiply-accumulate (MAC for polynomial multiplication) and degree reduction (DEGRED for polynomial module operation) subarrays. The salient feature of this digit-serial approach is that only the digit-cells are implemented in hardware, the finite field multiplications are performed digit-serially in software by dynamically scheduling the internal digit-level operations in RS encoders and decoders. It is concluded that, for 2-error-correcting RS(n,k) codec implementations over finite field GF(2/sup 8/), a parallel MAC unit (of digit-size 8) and a DEGRED unit with digit-size 2 is the best datapath, with respect to least energy consumption and energy-delay products. With this datapath architecture and appropriate digit-serial scheduling strategies, more than 60% energy reduction and more than 1/3 energy delay reduction can be achieved compared with the parallel multiplication datapath based approach.

This paper describes large-scale-integration programmable processors designed for multimedia processing such as real-time compression and decompression of audio and video as well as the generation of computer graphics. As the target of these processors is to handle audio and video in real time, the processing capability must be increased tenfold compared to that of conventional microprocessors, which were designed to handle mainly texts, figures, tables, and photographs. To clarify the advantages of a high-speed multimedia processing capability, we define these chips as multimedia processors. General-purpose microprocessors for workstations and personal computers (PCs) use special built-in hardware for multimedia processing, so the multimedia processors described include these modified general-purpose microprocessors. After reviewing the history of programmable processors, we classify multimedia processors into five categories depending on their basic architecture. The categories are reduced instruction set computer (RISC) microprocessors for workstations, complex instruction set computer microprocessors for PCs, embedded RISCs, low-power digital signal processors (DSPs), which are mainly used for mobile communications devices, and media processors that support PCs for multimedia applications. These five classes are then grouped into two: microprocessors with a multimedia instruction set and highly parallel DSPs. An architectural comparison between these two groups on the basis of Moving Picture Experts Group decoding applications is made, and the advantages and disadvantages of each class are clarified. Future processors, including "system on a chip," and their applications are also discussed.

A folded architecture and a digit-serial architecture are proposed for implementation of one- and two-dimensional discrete wavelet transforms. In the one-dimensional folded architecture, the computations of all wavelet levels are folded to the same low-pass and high-pass filters. The number of registers in the folded architecture is minimized by the use of a generalized life time analysis. The converter units are synthesized with a minimum number of registers using forward-backward allocation. The advantage of the folded architecture is low latency and its drawbacks are increased hardware area, less than 100% hardware utilization, and the complex routing and interconnection required by the converters used. These drawbacks are eliminated in the alternate digit-serial architecture at the expense of an increase in the system latency and some constraints on the wordlength. In latency-critical applications, the use of the folded architecture is suggested. If latency is not so critical, the digit-serial architecture should be used. The use of a combined folded and digit-serial architecture is proposed for implementation of two-dimensional discrete wavelet transforms.>

A building block for a scalable signal processor array is developed with a general-purpose DSP and a message routing LSI. Each DSP can be connected by multiple routing LSIs forming a point-to-point message-passing network with data packet communication. Low network latency is obtained by cut-through routing technique with sufficient communication bandwidth. The employment of an on-chip routing table allows regular as well as irregular topologies with complex routing techniques such as broad/multi-casting and dynamic routing. The combination of DSPs ( mu PD77240), a flexible message-passing network and an optional application-specific I/O interface makes the processor array suitable for a wide range of high speed signal processing applications such as adaptive array processing and 3-D vision processing.>

A software design system for asynchronous multirate/multitask processing is developed for a programmable digital signal processor, the NEC77240. A new scheduling method which combines static and dynamic scheduling is proposed. This avoids runtime overheads due to interrupts in asynchronous multirate system design for digital signal processors (DSPs). The processing delay is avoided by introducing deadline scheduling in the static scheduling. In this system, a block diagram description language is extended to describe a asynchronous multitask processing. By using this scheduling method, asynchronous multirate processing such as arbitrary sampling ratio rate conversion, asynchronous interface and multimedia applications can be efficiently realized by programmable DSPs.>

A VLSI chip set capable of real-time MPEG (Moving Picture Experts Group) video encoding/decoding has been developed. It is composed of an inter-frame prediction chip, a transform and quantization chip, and a variable length coding chip. To make the chip set more cost effective, the MPEG algorithms are first partitioned into three blocks on the basis of their characteristics. Individual chip architectures are designed with the use of programmable DSP and application specific array approaches. A hierarchical data transmission method is introduced for use among the chips and frame memories. By using three chips, an MPEG video encounter can compress a 30-frames/s image sequence of 352 pels*240 lines. A decoder can be constructed with two chips for the same sequence.>

A Hi-Fi audio codec with an improved adaptive transform coding (ATC) algorithm is presented using digital signal processors (DSPs). An audio signal with a 20 kHz bandwidth sampled at 48 kHz is coded at a rate of 128 kb/s. The algorithm utilizes adaptive block size selection, which is effective for preecho suppression. A modified discrete cosine transform (MDCT) with a simple window set is employed to reduce block boundary noise without decreasing the performance of transform coding. In addition, a fast MDCT calculation algorithm, based on a fast Fourier transform, is adopted. Weighted bit allocation is employed to quantize the transformed coefficients. The codec was realized by a multiprocessor system composed of newly developed DSP boards. Subjective tests with the codec show that the coding quality is comparable to that of compact disc signals.>

VLSI chips for luminance/chrominance (Y/C) signal separation and synthesis have been developed. Application-specific FIR filter structures and canonical signed-digit representation (CSR) multipliers used in the filters make it possible to develop compact high-speed VLSI chips. A silicon compiler, which employs the optimal FIR filter structures and supplies the optimal filter design faculty, has contributed to quick VLSI development. Y/C signal separation using four video FIR filters and Y/C signal synthesis using three video FIR filters are implemented on single chips by 1.2- mu m CMOS technology.>

A programmable real-time high-definition television (HDTV) signal processor (HD-VSP) has been developed. For conventional TV signals, a previously reported video signal processor (VSP) has introduced flexible software control capability based on subregional processing. In order to expand such flexibility for real-time HDTV signal processing, the HD-VSP employs eight VSP clusters and programmable time-expansion/compression units. An input HDTV signal is converted to eight time-expanded subregional signals to reduce their sampling rate to that of conventional TV signals. The converted signals are then processed by the eight clusters in the same manner as the VSP. Therefore, programs developed for conventional TV signals can be applied to HDTV with little modification. Processed signals obtained from the eight clusters are time-compressed and multiplexed to reconstruct an output HDTV signal. This HD-VSP has 16 component processors per cluster and is capable of 2.5 giga-operations/s. Several coder programs, including a discrete cosine transform coder and an intraframe differential pulse code modulation (PCM) coder, are developed to evaluate HDTV coding efficiency.>

The architecture is an expanded version of a previously reported video signal processor in which a number of parallel processor clusters can be combined in a tandem connection form or in a parallel connection form. The new video signal processor introduces programmable time-expansion and time-compression circuits to A-to-D and D-to-A converters, respectively, for coping with high speed HDTV signals. It also employs input/output switch units before and after parallel processor clusters. The introduction of input/output switch units to the parallel processor clusters makes it possible to input several video signals simultaneously. By these additional units, a HDTV signal is converted to a set of NTSC level video signals in the time-expansion circuit. Every NTSC level video signal is then delivered to parallel processor clusters through an input switch unit. After processing in clusters, NTSC level signals are converted to a HDTV signal through an output switch unit and time-compression circuits. This architecture can be applied to stereo scope processing. >

To increase efficiency on interframe coding and/or to preserve speaker privacy for videophone applications, a method for real-time background separation based on stereoscopic range finding is developed. The algorithm includes foreground center extraction, foreground threshold calculation, occlusion-free direct block matching with quadtree block subdivision, and contour segmentation using a motion edge. The algorithm is evaluated using a stereo-video sequence. For an application using the background separation technique, background filtering is introduced. Background filtering can decrease the number of inessential coded blocks, when applied to a motion-compensated hybrid differential pulse-code modulation (DPCM)/transform coder. If the cut-off frequency for filtering characteristics is further lowered, the filtering will conceal details in the background.>

A coding technique is presented for high-quality audio signals based on adaptive transform coding (ATC). Adaptive block size selection by the proposed algorithm ensures an appropriate block size resulting in improved SNR (signal-to-noise ratio) for a wide variety of source signals. A feedback approach, based on SNR, and a feedforward approach, based on interblock differences in input time-domain samples, to adaptive block size assignment are proposed and evaluated. Computer simulation results show that average segmental SNR by the feedback approach is improved by as much as 4.8 dB over the conventional fixed-block-size ATC. The feedforward approach is realized with much-simplified hardware; nevertheless, its SNR degradation from that by the feedback approach is 1.6 dB, even in the worst case. Both approaches are successful in pre-echo suppression to a satisfactory level. Time-domain aliasing cancellation has the potential to increase the superiority of the new algorithm.>

The authors present a single-chip VLSI chrominance/luminance (Y/C) separator that is economically fabricated for NTSC TV signals at 13.5-MHz CCIR standard sampling rate. In order to realize compactness and low power dissipation, two FIR filter architectures and a multiplier structure are proposed. A silicon compiler, which uses these structures, also contributes to fast and error-free VLSI development. The Y/C separator chip has 10.4-mm*11.7-mm die size and attains about 860-MOPS operating speed.>

A single-board 14.3-MOPS (million operations per second) video signal processor module (VSPM) has been developed. The module is fully microprogrammable and processes up to a 128 pel*128 pel subimage every 16.7 ms. Using a number of homogeneous VSPMs aligned in parallel, a real-time video signal processing environment is provided on the basis of an overlap-save or overlap-add technique. An experimental system has been constructed in order to demonstrate the signal processor approach's effectiveness for video signals by implementing picture coding algorithms. Due to software control capability, various kinds of picture coding techniques can be evaluated by the system.>

Two simple 64-kb/s wideband coding approaches using 32-kb/s ADPCM (adaptive digital pulse-code modulated) channel banks are proposed and compared to CCITT 64 kb/s ADPCM, which is being recommended as CCITT G.722. These two, folding ADPCM and QMF ADPCM, are intended to pave the way for smooth transition from conventional 4-kHz band telephone systems to 7-kHz wideband systems in private networks. The first approach, supporting the high-quality audio program transmission, requires only samplers and multiplexers at the input and output ports of the channel banks. In the second approach, samplers and multiplexers are replaced by quadrature mirror filters in order to increase coding quality. Performance test results for audio signal transmission show that these simplified approaches provide an inexpensive way to introduce wideband communication systems.>

Two important functional LSIs for the realtime Video Signal Processor (VSP) have been developed. One is the Pipelined Arithmetic Unit (PAU) and the other is the Address Generation Unit (AGU). The PAU chip employs a flexible pipelined architecture optimized for L1 or L2-norm distance calculation, used in a wide variety of image processing. The AGU chip, including 15-word register file for pointer alteration, offers a user-friendly two dimensional pointer addressing. A realtime Video Signal Processor Module (VSPM), composed of one PAU, four AGUs and memories, has been implemented for multiprocessor VSP configuration. Thanks to software control capability, various kinds of picture coding techniques can be evaluated by the system.

A simple programming system has been developed for a 32 bit floating point signal processor µPD77230 [1]. The system is based on the blockdiagram language concept [2][3], in which many signal processing systems can be realized by combinations of primitive functional blocks. A wide dynamic range and high precision processing obtained from a 32 bit floating point arithmetic make this approach possible, without complex scaling or limit cycle oscillation problems encountered in a fixed point processing. About 100 primitive programs are implemented on a mask ROM sample chip. They can be used as library programs accessed by a main program stored in an external instruction memory. In addition, library programs can also be carried out by a command sequence stored in an internal memory under a special control program in the chip. This approach enables using µPD77230 processing capability without knowledge concerning its architecture or a set of microinstructions. In this paper, a DTMF receiver and an ADPCM coder, realized by this system, are shown as examples.

A CMOS single chip signal processor, which has 32-bit floating point arithmetic units and large capacity on-chip memories, has been developed. The processor, having a floating point parallel multiplier, a floating point accumulator, two 512- word data RAMs, a 1024-word data ROM and a 2048- word program ROM, is implemented within a 15.4 × 8.4 mm chip area, containing 370,000 elements. As the processor is designed to perform highly accurate multiply-accumulate operations for digital filtering and to attain complex addressing capability for FFT computation, this processor can execute FIR computation at the 150 nsec per tap rate, as well as achieve 1024 point complex FFT computation in 12.3 msec.

A real-time video signal processor (VSP), suitable for VLSI implementation, is proposed. In order to obtain software controllability for video signal processing in a real-time environment, the VSP employs a multiprocessor configuration, where a plural number of video signal processing modules (VSPMs) are connected to input, output and feedback buses. Each VSPM processes its own assigned sub-images without communicating from/to other VSPMs. As the VSP needs no overhead for VSPM communication, the number of VSPMs involved does not affect the VSP system performance. The homogeneous processor based approach, employed in VSP, will become the most suitable LSI implementation methodology in the near future, due to processor module regularity and system flexibility.

A simple and universal PCM/ADPCM transcoding algorithm, which does not accumulate distortion in multiple code conversions between PCM and ADPCM, has been derived by thoroughly investigating the PCM-ADPCM-PCM code conversion process. Conditions to avoid distortion in code conversions are clarified by newly introduced mathematical expressions for ADPCM and PCM quantization processes. These expressions can be utilized to simplify a code conversion process without losing generality. Distortion reduction on a code conversion can be realized by adding some functions to the ADPCM decoder output portion. This distortion reduction technique is shown to assure no distortion accumulation in tandem code conversions. In contrast to other reported techniques for tandem conversions, the proposed method is free from applied ADPCM structures, as well as being the simplest among them. As the mathematical treatment is so general, this technique can be applied to code conversions other than PCM/ADPCM conversions.

An LSI ADPCM codec, which is based on the CCITT standard 32 kbps algorithm, has been developed. The LSI chip has been designed as a software controllable signal processor whose architecture is optimized for the CCITT algorithm. A reconfigurable pipeline multiplier-normalizer-accumulator circuit is effectively utilized for realizing complex ADPCM specifications. The LSI chip, implemented by 2.5 µ CMOS technology, dissipates only 90 milliwatts of power.

An ADPCM codec, that can provide toll quality speech at a 32 kb/s transmission rate, has been implemented on a single chip signal processor. Maximum effort has been paid to design a robust adaptation scheme for a quantizer and a predictor to withstand transmission bit errors. The codec employs a simplified robust quantizer and also employs a new backward adaptive predictor. The decoder, including the new adaptive predictor, has a structure having fixed poles and adaptive zeros, attaining both high prediction capability and robustness. The performance of a developed codec, which has analog interface capability through a PCM codec chip, satisfies the standard 64 kb/s PCM performance specification in CCITT recommendation G.712.

The development of a microprocessor-controlled 96/48 DSI terminal is described. The terminal is intended for domestic use, and is designed so as to maximize compatibility with existing digital carrier systems, it merges four T1 signals into two T1 compatible bit streams for transmission. The terminal is given sufficient excess capacity to accommodate voiceband data, while retaining toll-grade transmission quality for speech. Implementation approaches and considerations in both calculated and measured system characteristics are described.

A single chip, versatile digital signal processor has been developed, which enables compact and low cost telecommunications equipment realization. The processor architecture has been optimized for real time voiceband signal processing by adopting efficient micro-program control and by integrating a 16 × 16 bit parallel multiplier on a single chip. The processor is fabricated with N-channel MOS technology, and operates at a clock speed up to more than 8 MHz. Any instruction, including multiplication, can be performed within a 250 nanosecond interval. The paper also describes applications to actual voiceband communications equipment to prove the processor effectiveness.