This paper studies the statistical behavior of an affine combination of the outputs of two LMS adaptive filters that simultaneously adapt using the same white Gaussian input. The purpose of the combination is to obtain an LMS adaptive filter with fast convergence and small steady-state mean-square error (MSE). The linear combination studied is a generalization of the convex combination, in which the combination factor is restricted to the interval (0,1). The viewpoint is taken that each of the two filters produces dependent estimates of the unknown channel. Thus, there exists a sequence of optimal affine combining coefficients which minimizes the MSE. The optimal unrealizable affine combiner is studied and provides the best possible performance for this class. Then, a new scheme is proposed for practical applications. It is shown that the practical scheme yields close-to-optimal performance when properly designed (as suggested by the theoretical optimal).

This paper presents a novel implementation for identifying sparse telephone network echo channels. The new scheme follows the approach used in [1] in that the location of the channel response peak is estimated in the wavelet domain. A short time-domain adaptive filter is then located about the estimated peak to identify the sparse response. The primary purpose of this paper is to present an efficient design of such system. The use of a new block wavelet transform results in both 70% less computational complexity and improved peak detection. A new robust time-domain adaptive filtering is also proposed which significantly reduces the jitter problem in [1]. Monte Carlo simulations show excellent echo cancellation for a typical ITU-T channel.

This paper studies the behavior of the LMS algorithm for a special system identification problem when partial wavelet transformations restrict the algorithm's input vector to a subspace of the unknown system's input vector space. It is shown that the independence theory is not applicable in this case. A new theoretical model for the weight mean and fluctuation behaviors is developed which incorporates the correlation between successive data vectors (as opposed to using the independence theory model). Comparison of the new model predictions with Monte Carlo simulations shows good-to-excellent agreement, certainly much better than predicted by the independence theory model.

Echo cancellers are in wide use in both electrical (four wire to two wire mismatch) and acoustic (speaker-microphone coupling) applications. One of the main design problems is the control logic for adaptation. Basically, the algorithm weights should be frozen in the presence of double-talk and adapt quickly in the absence of double-talk. The control logic can be quite complicated (C. Breining et al., 1999) since it is often not easy to discriminate between the echo signal and the near end-speaker. This paper derives a log likelihood ratio test for deciding between double-talk (freeze weights) and a channel change (adapt quickly) using a stationary Gaussian stochastic input signal model. The probability density function of a sufficient statistic under each hypothesis is obtained and the performance of the test is evaluated as a function of the system parameters. The receiver operating characteristics indicate that it is difficult to correctly decide between double-talk and a channel change based upon a single look. However, post-detection integration of approximately one hundred sufficient statistic samples yields a detection probability close to unity with a small false alarm probability

The paper presents an analytical model for predicting the stochastic behavior of the affine projection (AP) algorithm operating in a nonstationary environment. The model is derived for autoregressive (AR) Gaussian inputs and for unity step size (fastest convergence). Deterministic recursive equations are presented for the mean weight and mean square error for a large number of adaptive taps, N, as compared to the algorithm order, P. The model predictions show excellent agreement with Monte Carlo simulations in transient and steady-state. The learning behavior of the AP algorithm in nonstationary environments is of great interest in applications such as acoustic echo cancellation.

The paper presents a novel scheme for identifying the impulse response of a sparse channel. The scheme consists of two adaptive filters operating sequentially. The first adaptive filter adapts using a partial Haar transform of the input and yields an estimate of the location of the peak of the sparse impulse response. The second adaptive filter is then centered about this estimate. Both filters are short in comparison to the delay uncertainty of the unknown channel. The principle advantage of this scheme is that two short adaptive filters can be used instead of one long adaptive filter, resulting in faster overall convergence and reduced computational complexity and storage.

The paper presents an analytical model for predicting the stochastic behavior of the affine projection (AP) algorithm. The model is derived for autoregressive (AR) Gaussian inputs and for unity step size (fastest convergence). Deterministic recursive equations are presented for the mean weight and mean square error for a large number of adaptive taps, N, as compared to the algorithm order, P. The model predictions show better agreement between theory and simulations in transient and steady-state than previous models described in the literature. The learning behavior of the AP algorithm is of great interest in applications such as acoustic echo cancellation.

This paper studies the statistical behavior of the affine projection (AP) algorithm for /spl mu/=1 for Gaussian autoregressive inputs. This work extends the theoretical results of Rupp (1998) to the numerical evaluation of the MSE learning curves for the adaptive AP weights. The MSE learning behavior of the AP(P+1) algorithm with an AR(Q) input (Q/spl les/P) is shown to be the same as the NLMS algorithm (/spl mu/=1) with a white input with M-P unity eigenvalues and P zero eigenvalues and increased observation noise. Monte Carlo simulations are presented which support the theoretical results.

This paper investigates the properties of the performance surface for a problem of nonlinear mean-square estimation of a random sequence. The problem studied has direct application to the study of active noise control (ANC) systems when the transducers are driven into nonlinear behavior. A deterministic expression is derived for the mean-square error (MSE) surface as a function of the system's degree of nonlinearity. It is shown how the presence of the nonlinearity deforms the MSE surface. It is demonstrated that the surface is unimodal, and the expression for the optimum weight vector is determined. The new results are then used to quantify the behavior of ANC systems employing the LMS adaptive algorithm. Important algorithm properties are derived from this study. Examples are presented to verify the analytical models derived.

This paper presents a stochastic analysis of the delayed least mean square (DLMS) adaptive algorithm using a new model. The new model does not use independence theory. Recursive difference equations are derived for the weight vector first and second moments. These equations yield new analytical results for the mean square error behavior. These results are compared to those of previous models. The new model is shown to be more general. The algorithm's properties are explained that could not be explained using existing models. The theoretical behavior is in close agreement with Monte Carlo simulations for the cases studied. This provides support for the accuracy of the theoretical model.

This paper presents a statistical analysis of the multiple-error Filtered-X Least Mean Square algorithm with nonlinearities in the secondary paths. Each nonlinearity is modelled by a scaled error function that can represent the saturation behavior of the transducers and associated hardware in active noise control applications. Deterministic recursions are derived for the mean weight and mean square error behavior for white Gaussian inputs and slow adaptation. The independence theory is not used in the derivations. Monte Carlo simulations show excellent agreement with the behavior predicted by the theoretical models. The new results predict the behavior of the algorithm for different amounts of nonlinear distortion.

This paper presents analytical, numerical and experimental results for a stochastic gradient adaptive scheme which identifies a polynomial-type nonlinear system with memory for noisy output observations. The analysis includes the computation of the stationary points, the mean square error surface, and the mean behaviour of the algorithm for Gaussian data. Monte Carlo simulations confirm the theoretical predictions which show a small sensitivity to the observation noise.

This paper presents a statistical analysis of the least mean square (LMS) algorithm when a zero-memory nonlinearity appears at the adaptive filter output. The nonlinearity is modelled by a scaled error function. Deterministic nonlinear recursions are derived for the mean weight and mean square error (MSE) behavior for white Gaussian inputs and slow adaptation. Monte Carlo simulations show excellent agreement with the behavior predicted by the theoretical models. The analytical results show that a small nonlinear effect has a significant impact on the converged MSE.

This paper presents a new analytical model for the second moment behavior of the filtered-X LMS algorithm. The new model is not based on the independence theory, and is derived for Gaussian inputs and slow adaptation. Monte Carlo simulations show excellent agreement with the behavior predicted by the theoretical model.

Gradient search adaptive algorithm for identifying an unknown nonlinear system comprised of a discrete-time linear system H followed by a zero-memory nonlinearity g(.). The LMS algorithm first estimates H. The weights are then frozen. Recursions are derived for the mean and fluctuation behavior of LMS which agree with Monte Carte simulations. When the nonlinearity is modelled by a scaled error function, the second part of the gradient scheme is shown to correctly learn the scale factor and the error function scale factor. Mean recursions for the scale factors show good agreement with Monte Carlo simulations.

Feed-forward multi-layer neural networks (MLNNs) are complex nonlinear learning systems which can be trained by well-known rules such as back-propagation (BP). The resulting adaptation procedures are extremely difficult to analyze for stochastic training data. Significant analytic results have been obtained for the single-layer case and for some simple two-layer cases. Previously, a structural simplification has been studied which models each threshold function as a linear device. This linearized MLNN can only create hyperplane decision rules after convergence. However, the multiplicative behavior of the layers may offer some performance advantages over linear adaptive algorithms (LMS or RLS) when used for a linear problem. A new log-domain linear MLNN adaptive structure is proposed and analyzed here. The log operation converts the layer multiplications into additions whereupon linear analysis techniques can be used. The transient and steady-state statistical behavior of the log linear MLNN is analyzed for Gaussian training data. Deterministic recursions are derived for the mean and fluctuation behavior of the new algorithm. These recursion are shown to be in excellent agreement with Monte Carlo simulations.

This paper presents a stochastic analysis of the Filtered-X LMS algorithm. The mean weight vector recursion is derived for slow adaptation and for a white reference signal without use of independence theory. The Wiener solution is determined explicitly as a function of the input statistics and the impulse responses of the primary and secondary signal paths. It is shown that the steady-state mean weights for the Filtered-X LMS algorithm converge to the Wiener solution only if the estimate of the secondary path is without error. Monte Carlo simulations show excellent agreement with the behavior predicted by the theoretical model.

This paper presents a neural network approach for modeling nonlinear memoryless communication channels. In particular, the paper studies the approximation of the nonlinear characteristics of traveling-wave tube (TWT) amplifiers used in satellite communications. The modeling is based upon multilayer neural networks, trained by the odd and even backpropagation (BP) algorithms. Simulation results demonstrate that neural network models fit the experimental data better than classical analytical TWT models,.

This paper analyzes the perceptron mean weight learning behavior for a system identification model with Gaussian input training data and fixed non zero biases for both the perceptron and the unknown system. The analysis is based upon the partial evaluation of certain expectations using Price's (1958) theorem followed by numerical integration in the mean weight recursions. The mean weight vector is shown to be in the same direction as that of the unknown system. A scalar recursion is derived for the length of the mean weights. The recursion is shown to yield weight vector predictions that are in close agreement with Monte Carlo simulations of the perceptron learning behavior. The stationary points are also accurately predicted by the theoretical model.

The least mean squares (LMS) algorithm is one of the most popular algorithms for digital implementation of real-time high-speed adaptive filters. This paper presents a study of the quantization effects in the finite precision LMS algorithm with power-of-two step sizes. Nonlinear recursions are derived for the mean and second moment matrix of the weight vector about the Wiener weight for white Gaussian data models and small algorithm step size /spl mu/. The solutions of these recursions are shown to agree very closely with the Monte Carlo simulations during all phases of the adaptation process. A design curve is presented to demonstrate the use of the theory to select the number of quantizer bits and the adaptation step size /spl mu/ to yield desired transient and steady-state behaviors.

The statistical learning behavior of the single-layer backpropagation algorithm was analyzed using a system identification formulation for noise-free training data [Bershad et al. 1993]. Transient and steady-state results were obtained for the mean weight behavior, mean-square error (MSE), and probability of correct classification. The present paper extends these results to the case of noisy training data. Three new analytical results are obtained: 1) the mean weights converge to finite values even when the bias terms are zero, 2) the MSE is bounded away from zero, and 3) the probability of correct classification does not converge to unity. However, over a wide range of signal-to-noise ratios (SNRs), the noisy training data does not have a significant effect on the perceptron stationary points relative to the weight fluctuations. Hence, one concludes that noisy training data has a relatively small effect on the ability of the perceptron to learn the model weight vector F.

This paper summarizes the results of the simulations and analysis of the learning behavior of a simple two-layer perceptron for a nonlinear system identification problem. Although it is difficult to generalize results for nonlinear systems, the analysis may improve our understanding of neural network training. Numerous sub-optimum stationary points occur for this problem and cause difficulties in the correct identification of the unknown system. The sub-optimum convergence points occur in the saturation regions of the various nonlinearities or for pathological cases. The size of the region of suboptimal convergence points may be reduced by increasing the dimensionality of the input data vector. Also, the range for the rate parameter is computed and an improvement to backpropagation is suggested.

Rosenblatt's algorithm is a recursive method that is often used to adjust the weights of a single-layer perceptron. It is capable of partitioning the input signal space into two regions that are separated by a hyperplane boundary. Thus, when the values of the input signal are linearly separable, the algorithm will converge to a stable stationary point that yields zero mean-square error. The authors examine the stationary points of Rosenblatt's algorithm when the data are not linearly separable. A system identification model is used to generate the data. An expression is also derived for the probability of an incorrect classification of the output signal when the weights are converged at a stationary point.>

A significant drawback of the least mean square (LMS) algorithm is slow convergence speed when the input covariance matrix is ill-conditioned. Two structures are presented and studied for increasing the convergence speed for this case. The structures incorporate a prewhitening filter prior to the usual LMS adaptation. When the prewhitening filter is also adaptive the input to the LMS algorithm is nonstationary. An analysis of the coupling effect between the two adaptive algorithms show that the adaptive prewhitener has the capability of significantly speeding up to LMS adaptation as compared to a system without prewhitening. When the prewhitening filter is fixed (nonadaptive), the structure is shown to be equivalent to the filtered-X LMS algorithm. Stability conditions and transient means behavior are given in the time domain, in terms of the parameters of the pre-whitening filter.>

The authors present a statistical analysis of the steady-state and transient properties of the single-layer backpropagation algorithm for Gaussian input signals. It is based on a nonlinear system identification model of the desired response which is capable of generating an arbitrary hyperplane decision boundary. It is demonstrated that, although the weights grow unbounded, the mean-square error decreases towards zero. These results indicate that the algorithm, on average, quickly learns the correct hyperplane associated with the system identification model. However, the nature of the mean-square error and the corresponding performance surface are such that the perceptron is prevented from correctly classifying with probability one until the weights converge at infinity.>

Preliminary results for the evolution of the filter coefficients adapted by the CM (constant modulus) algorithm, assuming that the input consists of sinusoids with Rayleigh amplitude distribution in the presence of additive noise, are presented. The stationary points of the iteration and the convergence behavior are discussed assuming proper filter initialization (i.e. lock is preserved) for the particular case of a single sinusoid in noise. The convergence of the average CM algorithm weight vector is analyzed, leading to three coupled nonlinear difference equations.>

The authors study the capabilities of the exponentially weighted recursive-least-squares (RLS) and least-mean-squares (LMS) algorithms, when configured as adaptive predictors, to track a chirped sinusoid in white background noise. The lag and fluctuation behaviors of each of the algorithms are calculated, and their influence on the misadjustment error is determined. The optimum tracking parameters for each algorithm are evaluated. The misadjustment errors for these optimum values are compared as a function of the chirp rate psi , the SNR rho , and the number of filter taps M. It is shown that for sufficiently small psi , small rho , and M such that rho M>>1, the LMS algorithm is superior to RLS because it has a smaller lag.>

Analog implementations of the LMS (least-mean squares) and block LMS (BLMS) adaptive filtering algorithms have been shown to be equivalent with respect to adaptation speed and steady-state misadjustment errors. However, the BLMS algorithm offers significant reductions in computational speed due to block processing. Digital implementations of the two algorithms are compared with respect to finite word effects. The algorithm stalling phenomena is studied using Gaussian data and conditional expectation arguments. It is shown that the BLMS algorithm requires 1/2(log/sub 2/L-K) fewer bits for the same stalling behavior (L=block length and K lies between 0.2 and 1, depending on the precise definition of algorithm stalling). The LMS algorithm requires log/sub 2/L fewer bits than BLMS for the same level of saturation behavior (transient response) at algorithm initialization. Hence, overall the LMS algorithm requires 1/2(log/sub 2/L+K) fewer bits than the BLMS algorithm for the same saturation and stalling effects.>

Digital implementation of the LMS (least-mean-squares) adaptive line enhancer (ALE) introduces certain nonlinear effects. Feedback-error-signal saturation effects on the ALE adaption are investigated. A set of nonlinear coupled difference equations is derived by projecting the mean weight vector upon a set of orthogonal basis functions. These equations are used to study the transient behavior of ALE for the case of one and two sinusoids in broadband noise.>

The effects of the least-mean-squares adaptive line enhancer (ALE) weight misadjustment errors on the bit error rate are investigated for a direct-sequence spread-spectrum binary communication system in the presence of strong narrowband interference. The converged ALE weights are modeled as the parallel connection of a deterministic FIR (finite impulse response) filter and a random FIR filter. The statistics of the random filter are derived, assuming the output of the random filter to be primarily due to the jammer convolved with random filter weights. This output is shown to be nonGaussian and to cause significant error rate degradation in comparison to a Gaussian model. Error probability expressions are derived for the bit error rate, evaluated numerically, and compared to the corresponding error probabilities for a Gaussian model for the random filter output. For some typical system parameter values and error probabilities it is shown that the Gaussian model yields performance results that are too optimistic by several decibels.>

In this paper, the joint probability density function of the weight vector in LMS adaptation is studied for Gaussian data models. An exact expression is derived for the characteristic function of the weight vector at time n+1 conditioned on the weight vector at time n. The conditional characteristic function is expanded in a Taylor series and averaged over the unknown weight density to yield a first order partial differential-difference equation in the un-conditioned characteristic function of the weight vector. The equation is solved approximately for small values of the adaptation parameter. The weights are shown to be jointly Gaussian with time varying mean vector and covariance matrix given as the solution to well-known difference equations for the weight vector mean and covariance matrix.

The insertion of dither at the input to a quantizer is a well-known technique for extending the dynamic range of that operation. Employment of this technique requires that some method be used to reduce or eliminate the effects of the dither signal on the system. A common method is to shape the spectrum of the dither so that it will be rejected. In this paper, dither is used to "linearize" the rounding quantizer at each accumulator input of the vector complex LMS adaptation algorithm, and the effects of dither and quantization noise on steady-state mean square error are studied. It is shown that if the dither vector is temporally white, then the increase in steady-state mean square error due to dither is larger than that due to quantization noise, regardless of the spatial properties of the dither vector. However, it is also shown that the affects of dither can be made neglegible compared to that of quantization noise by properly shaping the temporal spectrum of the dither vector.

The transient and steady-state weight correlation statistics of both the real and complex LMS adaptive filters are obtained when the inputs are independent samples from real and circularly Gaussian processes respectively. A matrix relationship is derived between the covariance matrix of the weight vector at two different times and the covariance matrix of the weights at one time. These expressions show that the weight fluctuations have the same time constants as the mean behavior of the LMS algorithm itself (i.e., the weights are correlated over the same number of iterations that it takes for the algorithm to converge to the Wiener weights for stationary inputs).

In this paper, a study is made of the joint probability density function of the single weight for the complex scalar LMS adaptive algorithm. Integral equations are derived for the probability density functions and characteristic functions of the real and imaginary parts of the weights. These integral equations are approximately solved for a sufficiently small adaptation parameter µ. It is shown that, for small µ and, in steady-state, the real and imaginary parts of the weights are statistically independent Gaussian random variables with mean equal to the Wiener weight.

Quantization noise effects in a digital implementation of the complex scalar LMS adaptive algorithm are analyzed by modelling the accumulator input quantizer as a memoryless nonlinearity. The probability density function of the quantizer input, conditioned on the weight, is derived. The density is applied to finding the first and second conditional moments at the quantizer output. These are used to derive difference equations that approximate the dynamical behavior of the first and second weight moments. These difference equations, when solved numerically, compare favorably with simulation results.

The performance of a matched filter square-law envelope detector is investigated for a new signal-in-noise model, that of a partially coherent-fading random-phase sine wave in additive zero-mean noise. This model is of interest when analyzing the effects of random amplitude and/or phase channels on narrow-band input signals. The amplitude and the phase of the sine wave are assumed to be independent random processes with known bandwidths. This paper obtains deflection-criteria detection-performance results for the envelope detector as a function of the bandwidths of the amplitude and phase of the signal process, the noise process, the observation time, and the second-moment statistics of the amplitude, phase, and noise processes. The results show quantitatively that the detection performance of the envelope detector is much more sensitive to loss of phase coherence than to loss of amplitude coherence.

An analytical technique is presented here for obtaining approximate expressions for both the phase and message estimate variances for an FM demodulator. Numerical computations, for both above and below threshold operation, demonstrate the accuracy of the approximate expressions.

Two versions of a Gaussian signal, corrupted by independent Gaussian noises, are available for estimating the signal-to-noise power ratio. This correspondence demonstrates that the sample correlation coefficient can be used to construct an unbiased estimator that is asymptotically efficient.

The problem of eliminating intersymbol interference is studied from the viewpoint of channel-state estimation. The channel state vector, at the beginning of the present message baud, contains the information required to eliminate the effects of past channel inputs. A receiver is proposed that makes a maximum-likelihood estimate of the channel state vector conditioned upon all previous digits having been received correctly. The channel-state estimate is used to control the motion of a variable decision threshold. Correct adjustment of the threshold removes the effects of past symbols on the present decision. The receiver performance is studied via certain approximation techniques used in the study of threshold learning systems. In particular, the performance of one- and two-pole channels is evaluated. In these cases, the receiver is shown to effectively eliminate intersymbol interference and perform as well as a matched-filter receiver that uses only the signal energy received during one message interval.

An approach to the analysis of threshold learning suggested by the classical theory of Brownian motion is presented. In particular, it is shown how a nonlinear Langevin equation represents the motion of the threshold of a trainable signal detector, and how a Fokker-Planck diffusion equation yields an estimate of the shape of the probability density of the threshold. Our results are applicable to all trainable signal detectors in which the training procedure 1) raises the threshold in response to a false alarm, lowers the threshold in response to a false rest, and keeps the threshold unchanged in response to a correct decision, and 2) adjusts the size of the threshold increment by an amount that depends only on the trial number, and such that the threshold can eventually reach any real number.