Open-source relational database systems have become increasingly popular in the cloud era. However, practitioners are often beset with query performance issues. Thus a general-purpose database performance tuning tool independent of the various DBMS kernels becomes desired to lower the bar of using these systems. The first mandatory step in developing such a tool is to design an effective sampling method that collects representative records from different tables. Although one could leverage standard SQL statements and indexes to achieve this, sampling performance and statistical efficiency are not guaranteed when the underlying tables are frequently updated, especially for Sparse Tables where the range of index values is significantly greater than the table size.To this end, we propose a novel Active Sampling algorithm that queries regions more likely to contain data records from Sparse Tables. It relies on Gaussian process regression to characterize the probability density of whether a data record is non-null at a given index value. With the help of this estimated density function, the proposed method achieves efficient sampling by actively querying records with adaptive resolutions of interval lengths and provides an unbiased estimator for histogram construction. Comprehensive experiments on synthetic and real-world datasets demonstrate that the proposed Active Sampling method can effectively improve the estimation accuracy and use less query cost than other commonly used sampling methods.

Genome-wide association studies (GWAS) using longitudinal phenotypes collected over time is appealing due to the improvement of power. However, computation burden has been a challenge because of the complex algorithms for modeling the longitudinal data. Approximation methods based on empirical Bayesian estimates (EBEs) from mixed-effects modeling have been developed to expedite the analysis. However, our analysis demonstrated that bias in both association test and estimation for the existing EBE-based methods remains an issue. We propose an incredibly fast and unbiased method (simultaneous correction for EBE, SCEBE) that can correct the bias in the naive EBE approach and provide unbiased P-values and estimates of effect size. Through application to Alzheimer's Disease Neuroimaging Initiative data with 6 414 695 single nucleotide polymorphisms, we demonstrated that SCEBE can efficiently perform large-scale GWAS with longitudinal outcomes, providing nearly 10 000 times improvement of computational efficiency and shortening the computation time from months to minutes. The SCEBE package and the example datasets are available at https://github.com/Myuan2019/SCEBE.

MOTIVATION: It has been claimed in the literature that pooling DNA samples is efficient in estimating haplotype frequencies. There is, however, no theoretical justification based on calculation of statistical efficiency. In fact, the limited evidence given so far is based on simulation studies with small numbers of loci. With rapid advance in technology, it is of interest to see if pooling is still efficient when the number of loci increases. METHODS: Instead of resorting to simulation studies, we make use of asymptotic statistical theory to perform exact calculation of the efficiency of pooling relative to no pooling in the estimation of haplotype frequencies. As an intermediate step, we use the log-linear formulation of the haplotype probabilities and derive the asymptotic variance-covariance matrix of the maximum likelihood estimators of the canonical parameters of the log-linear model. RESULTS: Based on our calculations under linkage equilibrium, pooling can suffer huge loss in efficiency relative to no pooling when there are more than three independent loci and the alleles are not rare. Pooling works better for rare alleles. In particular, if all the minor allele frequencies are 0.05, pooling maintains an advantage over no pooling until the number of independent loci reaches 6. High linkage disequilibrium effectively reduces the number of independent loci by ruling out certain haplotypes from occurring. Similar calculations of efficiency for the case of no pooling justify the common belief that it is not worthwhile to use molecular methods to resolve the phase ambiguity of individual genotype data. AVAILABILITY: The R codes for the calculation are available at http://www.stat.nus.edu.sg/∼staxj/pooling CONTACT: [email protected].