Machine learning (ML) systems often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification in ML pipelines as a key reason for these failures. An ML pipeline is the full procedure followed to train and validate a predictor. Such a pipeline is underspecified when it can return many distinct predictors with equivalently strong test performance. Underspecification is common in modern ML pipelines that primarily validate predictors on held-out data that follow the same distribution as the training data. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We provide evidence that underspecfication has substantive implications for practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

The JMLR study Do we need hundreds of classifiers to solve real world classification problems? benchmarks 179 classifiers in 17 families on 121 data sets from the UCI repository and claims that “the random forest is clearly the best family of classifier”. In this response, we show that the study's results are biased by the lack of a held-out test set and the exclusion of trials with errors. Further, the study's own statistical tests indicate that random forests do not have significantly higher percent accuracy than support vector machines and neural networks, calling into question the conclusion that random forests are the best classifiers.

In this paper, we provide an introduction to machine learning tasks that address important problems in genomic medicine. One of the goals of genomic medicine is to determine how variations in the DNA of individuals can affect the risk of different diseases, and to find causal explanations so that targeted therapies can be designed. Here we focus on how machine learning can help to model the relationship between DNA and the quantities of key molecules in the cell, with the premise that these quantities, which we refer to as cell variables, may be associated with disease risks. Modern biology allows high-throughput measurement of many such cell variables, including gene expression, splicing, and proteins binding to nucleic acids, which can all be treated as training targets for predictive models. With the growing availability of large-scale data sets and advanced computational techniques such as deep learning, researchers can help to usher in a new era of effective genomic medicine.

MOTIVATION: Picking peaks from experimental NMR spectra is a key unsolved problem for automated NMR protein structure determination. Such a process is a prerequisite for resonance assignment, nuclear overhauser enhancement (NOE) distance restraint assignment, and structure calculation tasks. Manual or semi-automatic peak picking, which is currently the prominent way used in NMR labs, is tedious, time consuming and costly. RESULTS: We introduce new ideas, including noise-level estimation, component forming and sub-division, singular value decomposition (SVD)-based peak picking and peak pruning and refinement. PICKY is developed as an automated peak picking method. Different from the previous research on peak picking, we provide a systematic study of the proposed method. PICKY is tested on 32 real 2D and 3D spectra of eight target proteins, and achieves an average of 88% recall and 74% precision. PICKY is efficient. It takes PICKY on average 15.7 s to process an NMR spectrum. More important than these numbers, PICKY actually works in practice. We feed peak lists generated by PICKY to IPASS for resonance assignment, feed IPASS assignment to SPARTA for fragments generation, and feed SPARTA fragments to FALCON for structure calculation. This results in high-resolution structures of several proteins, for example, TM1112, at 1.25 A. AVAILABILITY: PICKY is available upon request. The peak lists of PICKY can be easily loaded by SPARKY to enable a better interactive strategy for rapid peak picking.

Ultra-wideband (UWB) communication systems are used in indoor environments with dense multi-path characteristics. Therefore channel estimation has an important role in the receiver of these systems. A new approach for data-aided (DA) and non-data-aided (NDA) channel estimation is proposed, which is called the pulse compression (PC) method. This method is useful for UWB systems employing time-hopping pulse position modulation. The PC method requires only some basic operations such as sampling, overlap-add and finite impulse response filtering. The PC method, in both DA and NDA scenarios, in spite of its low complexity, outperforms the maximum-likelihood (ML) method in channel parameters estimation. The bit error rate (BER) of the DA method, in single-user scenario, performs as well as the ML method, and in multi-user scenario, in the worst case, there is only 0.5 dB loss compared with the ML method. In the case of NDA scenario, the proposed method outperforms the NDA-ML method, that is, in the single-user scenario about 4 dB gain at the BER of 10−3 is observed. In multi-user scenario, it outperforms significantly the NDA-ML method, and its performance loss in comparison with the perfect channel knowledge scenario is about 3 dB at the BER of 10−3.

As the UWB systems are designed to operate in environments having dense multipath characteristics (i.e., indoor environments) the channel estimation task is much more involved than the conventional systems. In this paper, we present a new blind channel estimation technique for Time-Hopping Pluse-Position-Modulation (TH-PPM) systems. Our proposed method is based on the Pulse Compression (PC) technique introduced in [1]. In spite of its low complexity, the proposed blind channel estimation method outperforms the Non-Data Aided Maximum Likelihood (NDA-ML) technique. Our simulation results show the promising gain of 4 dB around the Bit Error Rate (BER) of 10-3, compared to NDA-ML. The performance loss, compared to the Perfect Channel Knowledge (PCK) is only 1.5 dB at this BER.