While language models have exceptional capabilities at text generation, they lack a natural inductive bias for emitting numbers and thus struggle in tasks involving quantitative reasoning, especially arithmetic. One fundamental limitation is the nature of the cross-entropy (CE) loss, which assumes a nominal scale and thus cannot convey proximity between generated number tokens. In response, we here present a regression-like loss that operates purely on token level. Our proposed Number Token Loss (NTL) comes in two flavors and minimizes either the $\mathcal{L}_p$ norm or the Wasserstein distance between the numerical values of the real and predicted number tokens. NTL can easily be added to any language model and extend the CE objective during training without runtime overhead. We evaluate the proposed scheme on various mathematical datasets and find that it consistently improves performance in math-related tasks. In a direct comparison on a regression task, we find that NTL can match the performance of a regression head, despite operating on token level. Finally, we scale NTL up to 3B parameter models and observe improved performance, demonstrating its potential for seamless integration into LLMs. We hope to inspire LLM developers to improve their pretraining objectives and distribute NTL as a minimalistic and lightweight PyPI package *ntloss*: https://ibm.biz/ntl-pypi-repo. Development code for full paper reproduction is available separately.

Gene regulatory networks (GRNs) and gene co-expression networks (GCNs) allow genome-wide exploration of molecular regulation patterns in health and disease. The standard approach for obtaining GRNs and GCNs is to infer them from gene expression data, using computational network inference methods. However, since network inference methods are usually applied on aggregate data, distortion of the networks by demographic confounders might remain undetected, especially because gene expression patterns are known to vary between different demographic groups. In this paper, we present a computational framework to systematically evaluate the influence of demographic confounders on network inference from gene expression data. Our framework compares similarities between networks inferred for different demographic groups with similarity distributions obtained for random splits of the expression data. Moreover, it allows to quantify to which extent demographic groups are represented by networks inferred from the aggregate data in a confounder-agnostic way. We apply our framework to test four widely used GRN and GCN inference methods as to their robustness w. r. t. confounding by age, ethnicity and sex in cancer. Our findings based on more than $ {44000}$ inferred networks indicate that age and sex confounders play an important role in network inference for certain cancer types, emphasizing the importance of incorporating an assessment of the effect of demographic confounders into network inference workflows. Our framework is available as a Python package on GitHub: https://github.com/bionetslab/grn-confounders.