Leafy vegetables have a huge demand in subtropical global markets for its high nutritional value and low cost. Accurate estimation of traits like leaf nitrogen concentration and shoot biomass is critical for optimal fertilizer dosing, nutrient content assessment as well as phenotyping studies. Hand-held smartphone with high-density Light Detection and Ranging (LiDAR) systems capture huge amounts of three-dimensional (3D) plant structural and intensity information that can be used to estimate plant traits. Thus, we propose a semiautomatic proximal sensing approach to model plant nitrogen content and shoot biomass using the structural and intensity information acquired by a smartphone based LiDAR sensor. The performance of the models in estimating leaf nitrogen concentration and shoot dry-weight biomass was quantified on Chinese broccoli (Brassica oleracea) plants with prior knowledge of nitrogen dossing concentrations, and produced a minimal root mean squared error of 8.75 mg nitrogen per gram of dry weight, and 2.38 g, respectively. The potential of the proposed modelling approach to accurately estimate leaf nitrogen concentration and shoot dry-weight biomass, from smartphone LiDAR data is proven.

The Welch (lower) Bound on the mean square cross correlation between n unit-norm vectors f1, ..., fnin the m dimensional space (Rmor Cm), for n ≥ m, is a useful tool in the analysis and design of spread spectrum communications, compressed sensing and analog coding. Letting F = [f1|...|fn] denote the m-by-n frame matrix, the Welch bound can be viewed as a lower bound on the second moment of F, namely on the trace of the squared Gram matrix (F' F)2. We consider an erasure setting, in which a reduced frame, composed of a random subset of Bernoulli selected vectors, is of interest. We present the erasure Welch bound and generalize it to the d-th order moment of the reduced frame, for d = 2, 3, 4. We provide simple, explicit formulae for the generalized bound, which interestingly is equal to the d-th moment of Wachter's classical MANOVA distribution plus a vanishing term (as n goes to infinity with m/n held constant). The bound holds with equality if (and for d = 4 only if) F is an Equiangular Tight Frame (ETF). Hence, our results offer a novel perspective on the superiority of ETFs over other frames, and provide explicit characterization for their subset moments.

A low rank matrix X has been contaminated by uniformly distributed noise, missing values, outliers and corrupt entries. Reconstruction of X from the singular values and singular vectors of the contaminated matrix Y is a key problem in machine learning, computer vision and data science. In this paper we show that common contamination models (including arbitrary combinations of uniform noise, missing values, outliers and corrupt entries) can be described efficiently using a single framework. We develop an asymptotically optimal algorithm that estimates X by manipulation of the singular values of Y, which applies to any of the contamination models considered. Finally, we find an explicit signal-to-noise cutoff, below which estimation of X from the singular value decomposition of Y must fail, in a well-defined sense.

We consider the recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE) - (square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level σ, the optimal nonlinearity for MSE loss simply shrinks each data singular value y to √(y2-4nσ2) (or to 0 if y2, which compares favorably with optimally tuned hard thresholding and optimally tuned soft thresholding, providing guarantees of 3nrσ2and 6nrσ2, respectively. Our general method also allows one to evaluate optimal shrinkers numerically to arbitrary precision. As an example, we compute optimal shrinkers for the Schatten-p norm loss, for any p > 0.

We consider recovery of low-rank matrices from noisy data by hard thresholding of singular values, in which empirical singular values below a threshold λ are set to 0. We study the asymptotic mean squared error (AMSE) in a framework, where the matrix size is large compared with the rank of the matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. The AMSE-optimal choice of hard threshold, in the case of n-by-n matrix in white noise of level σ, is simply (4/√3)√nσ ≈ 2.309√nσ when σ is known, or simply 2.858 · ymedwhen σ is unknown, where ymedis the median empirical singular value. For nonsquare, m by n matrices with m ≠ n the thresholding coefficients 4/√3 and 2.858 are replaced with different provided constants that depend on m/n. Asymptotically, this thresholding rule adapts to unknown rank and unknown noise level in an optimal manner: it is always better than hard thresholding at any other value, and is always better than ideal truncated singular value decomposition (TSVD), which truncates at the true rank of the low-rank matrix we are trying to recover. Hard thresholding at the recommended value to recover an n-by-n matrix of rank r guarantees an AMSE at most 3 nrσ2. In comparison, the guarantees provided by TSVD, optimally tuned singular value soft thresholding and the best guarantee achievable by any shrinkage of the data singular values are 5 nrσ2, 6 nrσ2, and 2 nrσ2, respectively. The recommended value for hard threshold also offers, among hard thresholds, the best possible AMSE guarantees for recovering matrices with bounded nuclear norm. Empirical evidence suggests that performance improvement over TSVD and other popular shrinkage rules can be substantial, for different noise distributions, even in relatively small n.

We present a document classification system that employs lazy learning from labeled phrases, and argue that the system can be highly effective whenever the following property holds: most of information on document labels is captured in phrases. We call this property near sufficiency. Our research contribution is twofold: (a) we quantify the near sufficiency property using the Information Bottleneck principle and show that it is easy to check on a given dataset; (b) we reveal that in all practical cases---from small-scale to very large-scale---manual labeling of phrases is feasible: the natural language constrains the number of common phrases composed of a vocabulary to grow linearly with the size of the vocabulary. Both these contributions provide firm foundation to applicability of the phrase-based classification (PBC) framework to a variety of large-scale tasks. We deployed the PBC system on the task of job title classification, as a part of LinkedIn's data standardization effort. The system significantly outperforms its predecessor both in terms of precision and coverage. It is currently being used in LinkedIn's ad targeting product, and more applications are being developed. We argue that PBC excels in high explainability of the classification results, as well as in low development and low maintenance costs. We benchmark PBC against existing high-precision document classification algorithms and conclude that it is most useful in multilabel classification.

MOTIVATION: Cloning of long DNA sequences (40-60 bases) into phage display libraries using polymerase chain reaction (PCR) is a low efficiency process, in which PCR is used to incorporate a DNA insert, coding for a certain peptide, into the amplified sequence. The PCR efficiency in this process is strongly affected by the distribution of G-C bases in the amplified sequence. As any DNA insert coding for the target peptide may be attempted, there is a flexibility in choosing part of the amplified sequence. Since the number of inserts coding for the same peptide is exponential in the peptide length, a computational problem naturally arises--that of efficiently finding an insert, whose parameters are optimal for PCR cloning. RESULTS: The GC distribution requirements are formulated as a search problem. We developed an efficient, linear time 'one pass' algorithm for this problem. Interestingly, our algorithm strongly relies on an interesting symmetry, which we observed in the standard genetic code. Most non-standard genetic codes examined possess this symmetry as well, yet some do not. We generalize the search problem and consider the case of a non-standard, or arbitrary, genetic code where this symmetry does not necessary hold. We solve the generalized problem in polynomial, but nonlinear, time. AVAILABILITY: An implementation of the proposed algorithm is available upon request from the authors.