We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text. Past work found that models are miscalibrated, with entropy per step increasing as generations grow longer, due to error accumulation. To calibrate the model and improve text quality, it has become standard practice to truncate the distribution, but this approach reduces output diversity, which we would like to avoid. Therefore, in this paper, we ask: does miscalibration improve automatically with scale, and if not, is it theoretically possible to calibrate without tradeoffs? To build intuition, we first study a simplified theoretical setting to characterize the scaling behavior of miscalibration with respect to dataset size. We find that the rate of scaling depends on the power law exponent of the data distribution --- in particular, for a power law exponent close to 1, the scaling exponent is close to 0, meaning that miscalibration improves very slowly with scale. Next, we measure miscalibration empirically in language models ranging from 0.5B to 70B parameters. We find that the observed scaling behavior is similar to what is predicted theoretically: our fitted scaling exponents for text are close to 0, meaning that larger models accumulate error at a similar rate as smaller ones. This scaling (or, lack thereof) provides one explanation for why we sample from larger models with similar amounts of truncation as smaller models, even though the larger models are of higher quality. However, truncation is not a satisfying solution because it comes at the cost of increased log loss. In theory, is it even possible to reduce entropy while preserving log loss? We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.

Relatedness within genomic cohorts is a potential source of bias for many genetic analyses. Existing tools to break relatedness are phenotype-naïve; they indiscriminately remove subjects to break relationship, risking the loss of valuable data, especially in studies targeting uncommon and rare phenotypes. To address this limitation, we developed the Kinship Decouple and Phenotype Selection (KDPS) tool, with a novel algorithm designed to enhance the precision of subject selection in genetic and epidemiological research by incorporating phenotype prioritization. KDPS separates related individuals by considering relatedness (kinship or identity by descent) scores and allows prioritizing subjects based on phenotypes of interest. This approach enables the retention of valuable subjects for analysis, even in the face of necessary exclusions due to relatedness. Furthermore, KDPS accommodates a wide range of phenotypes, including quantitative and categorical, and allows for customization to either prioritize specific phenotypes or maximization of sample size. In simulations based on the UK Biobank dataset and real-world datasets, KDPS demonstrated significant improvements in retention of subjects with prioritized phenotypes and computational efficiency compared to previously published software. The ability of this method to process biobank-scale studies within practical timeframes marks the ability of this method to process biobank-scale studies within practical timeframes and marks a considerable advancement over existing techniques. KDPS offers tailored solutions for complex analytical challenges and broad applicability in genetics and epidemiology research. To our knowledge, KDPS is the first tool to perform phenotype-aware decoupling, paving the way for more powerful genetic and epidemiological analyses.

Given only a few observed entries from a low-rank matrix $X$, matrix completion is the problem of imputing the missing entries, and it formalizes a wide range of real-world settings that involve estimating missing data. However, when there are too few observed entries to complete the matrix, what other aspects of the underlying matrix can be reliably recovered? We study one such problem setting, that of ``one-sided'' matrix completion, where our goal is to recover the right singular vectors of $X$, even in the regime where recovering the left singular vectors is impossible, which arises when there are more rows than columns and very few observations. We propose a natural algorithm that involves imputing the missing values of the matrix $X^TX$ and show that even with only two observations per row in $X$, we can provably recover $X^TX$ as long as we have at least $\Omega(r^2 d \log d)$ rows, where $r$ is the rank and $d$ is the number of columns. We evaluate our algorithm on one-sided recovery of synthetic data and low-coverage genome sequencing. In these settings, our algorithm substantially outperforms standard matrix completion and a variety of direct factorization methods.

Susceptibility tensor imaging (STI) is a promising tool for studying orientation-dependent tissue magnetic susceptibility and for mapping white matter fiber orientations complementary to diffusion tensor imaging (DTI). However, the limited head rotation range within modern head coils for data acquisition makes in vivo STI reconstruction ill-conditioned. Conventional STI reconstruction method is usually vulnerable to noise and requires sufficiently large head rotations to solve this ill-conditioned inverse problem. In this study, based on the recently proposed asymmetric STI (aSTI) model, a new method termed aSTI+ was proposed to improve in vivo STI reconstruction by enforcing isotropic susceptibility tensor inside cerebrospinal fluid (CSF) and applying morphology constraint in white matter. Experimental results showed superior performance of the proposed method with reduced noise, improved tissue contrast and better fiber orientation estimation over previous methods. Thus aSTI+ may promote in vivo human brain STI studies on white matter and myelin-related brain diseases.

We propose a method for unsupervised parsing based on the linguistic notion of a constituency test.One type of constituency test involves modifying the sentence via some transformation (e.g.replacing the span with a pronoun) and then judging the result (e.g.checking if it is grammatical).Motivated by this idea, we design an unsupervised parser by specifying a set of transformations and using an unsupervised neural acceptability model to make grammaticality decisions.To produce a tree given a sentence, we score each span by aggregating its constituency test judgments, and we choose the binary tree with the highest total score.While this approach already achieves performance in the range of current methods, we further improve accuracy by fine-tuning the grammaticality model through a refinement procedure, where we alternate between improving the estimated trees and improving the grammaticality model.The refined model achieves 62.8 F1 on the Penn Treebank test set, an absolute improvement of 7.6 points over the previous best published result.

We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.We first compare the benefits of no pre-training, fastText (Bojanowski et al., 2017;Mikolov et al., 2018), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018a) for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings.We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language.To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model.The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate.We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages.Finally, we demonstrate new state-ofthe-art results for 11 languages, including English (95.8 F1) and Chinese (91.8 F1).