Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, Hannaneh Hajishirzi. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

LLMs process text as sequences of tokens that roughly correspond to words, where less common words are represented by multiple tokens.However, individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise.For example, Llama-2-7b's tokenizer splits the word "northeastern" into the tokens [_n, ort, he, astern], none of which correspond to semantically meaningful units like "north" or "east."Similarly, the overall meanings of named entities like "Neil Young" and multi-word expressions like "break a leg" cannot be directly inferred from their constituent tokens.Mechanistically, how do LLMs convert such arbitrary groups of tokens into useful higher-level representations?In this work, we find that last token representations of named entities and multi-token words exhibit a pronounced "erasure" effect, where information about previous and current tokens is rapidly forgotten in early layers.Using this observation, we propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers, and present results of this method for Llama-2-7b and Llama-3-8b.To our knowledge, this is the first attempt to probe the implicit vocabulary of an LLM. 1

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 90\% confidence interval of around 2 to 22 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI [Formula: see text]). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI [Formula: see text]) are considered easier to register to T2w due to the availability of corresponding features. We propose a learning from privileged modality algorithm, using a training-only imaging modality DWI [Formula: see text], to support the challenging multi-modality registration problems. We present experimental results based on 369 sets of 3D multiparametric MRI images from 356 prostate cancer patients and report, with statistical significance, a lowered median target registration error of 4.34 mm, when registering the holdout DWI [Formula: see text] and T2w image pairs, compared with that of 7.96 mm before registration. Results also show that the proposed learning-based registration networks enabled efficient registration with comparable or better accuracy, compared with a classical iterative algorithm and other tested learning-based methods with/without the additional modality. These compared algorithms also failed to produce any significantly improved alignment between DWI [Formula: see text] and T2w in this challenging application.

The assessment of the quality of synthesised/pseudo Computed Tomography (pCT) images is commonly measured by an intensity-wise similarity between the ground truth CT and the pCT. However, when using the pCT as an attenuation map (μ-map) for PET reconstruction in Positron Emission Tomography Magnetic Resonance Imaging (PET/MRI) minimising the error between pCT and CT neglects the main objective of predicting a pCT that when used as μ-map reconstructs a pseudo PET (pPET) which is as similar as possible to the gold standard CT-derived PET reconstruction. This observation motivated us to propose a novel multi-hypothesis deep learning framework explicitly aimed at PET reconstruction application. A convolutional neural network (CNN) synthesises pCTs by minimising a combination of the pixel-wise error between pCT and CT and a novel metric-loss that itself is defined by a CNN and aims to minimise consequent PET residuals. Training is performed on a database of twenty 3D MR/CT/PET brain image pairs. Quantitative results on a fully independent dataset of twenty-three 3D MR/CT/PET image pairs show that the network is able to synthesise more accurate pCTs. The Mean Absolute Error on the pCT (110.98 HU ± 19.22 HU) compared to a baseline CNN (172.12 HU ± 19.61 HU) and a multi-atlas propagation approach (153.40 HU ± 18.68 HU), and subsequently lead to a significant improvement in the PET reconstruction error (4.74% ± 1.52% compared to baseline 13.72% ± 2.48% and multi-atlas propagation 6.68% ± 2.06%).

David Atkinson, Kumar Bhargav Srinivasan, Chenhao Tan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Direct reconstruction of parametric images from raw photon counts has been shown to improve the quantitative analysis of dynamic positron emission tomography (PET) data. However it suffers from subject motion which is inevitable during the typical acquisition time of 1-2 hours. In this work we propose a framework to jointly estimate subject head motion and reconstruct the motion-corrected parametric images directly from raw PET data, so that the effects of distorted tissue-to-voxel mapping due to subject motion can be reduced in reconstructing the parametric images with motion-compensated attenuation correction and spatially aligned temporal PET data. The proposed approach is formulated within the maximum likelihood framework, and efficient solutions are derived for estimating subject motion and kinetic parameters from raw PET photon count data. Results from evaluations on simulated [11C]raclopride data using the Zubal brain phantom and real clinical [18F]florbetapir data of a patient with Alzheimer's disease show that the proposed joint direct parametric reconstruction motion correction approach can improve the accuracy of quantifying dynamic PET data with large subject motion.

This work provides an insight into positron emission tomography (PET) joint image reconstruction/motion estimation (JRM) by maximization of the likelihood, where the probabilistic model accounts for warped attenuation. Our analysis shows that maximum-likelihood (ML) JRM returns the same reconstructed gates for any attenuation map (μ-map) that is a deformation of a given μ-map, regardless of its alignment with the PET gates. We derived a joint optimization algorithm accordingly, and applied it to simulated and patient gated PET data. We first evaluated the proposed algorithm on simulations of respiratory gated PET/CT data based on the XCAT phantom. Our results show that independently of which μ-map is used as input to JRM: (i) the warped μ-maps correspond to the gated μ-maps, (ii) JRM outperforms the traditional post-registration reconstruction and consolidation (PRRC) for hot lesion quantification and (iii) reconstructed gated PET images are similar to those obtained with gated μ-maps. This suggests that a breath-held μ-map can be used. We then applied JRM on patient data with a μ-map derived from a breath-held high resolution CT (HRCT), and compared the results with PRRC, where each reconstructed PET image was obtained with a corresponding cine-CT gated μ-map. Results show that JRM with breath-held HRCT achieves similar reconstruction to that using PRRC with cine-CT. This suggests a practical low-dose solution for implementation of motion-corrected respiratory gated PET/CT.

The combination of positron emission tomography (PET) and magnetic resonance imaging (MRI) offers unique possibilities. In this paper we aim to exploit the high spatial resolution of MRI to enhance the reconstruction of simultaneously acquired PET data. We propose a new prior to incorporate structural side information into a maximum a posteriori reconstruction. The new prior combines the strengths of previously proposed priors for the same problem: it is very efficient in guiding the reconstruction at edges available from the side information and it reduces locally to edge-preserving total variation in the degenerate case when no structural information is available. In addition, this prior is segmentation-free, convex and no a priori assumptions are made on the correlation of edge directions of the PET and MRI images. We present results for a simulated brain phantom and for real data acquired by the Siemens Biograph mMR for a hardware phantom and a clinical scan. The results from simulations show that the new prior has a better trade-off between enhancing common anatomical boundaries and preserving unique features than several other priors. Moreover, it has a better mean absolute bias-to-mean standard deviation trade-off and yields reconstructions with superior relative$\ell ^{2}$-error and structural similarity index. These findings are underpinned by the real data results from a hardware phantom and a clinical patient confirming that the new prior is capable of promoting well-defined anatomical boundaries.

The Magnetic Resonance Imaging (MRI) signal can be made sensitive to functional parameters that provide information about tissues. In dynamic contrast enhanced (DCE) MRI these functional parameters are related to the microvasculature environment and the concentration changes that occur rapidly after the injection of a contrast agent. Typically DCE images are reconstructed individually and kinetic parameters are estimated by fitting a pharmacokinetic model to the time-enhancement response; these methods can be denoted as "indirect". If undersampling is present to accelerate the acquisition, techniques such as kt-FOCUSS can be employed in the reconstruction step to avoid image degradation. This paper suggests a Bayesian inference framework to estimate functional parameters directly from the measurements at high temporal resolution. The current implementation estimates pharmacokinetic parameters (related to the extended Tofts model) from undersampled (k, t)-space DCE MRI. The proposed scheme is evaluated on a simulated abdominal DCE phantom and prostate DCE data, for fully sampled, 4 and 8-fold undersampled (k, t)-space data. Direct kinetic parameters demonstrate better correspondence (up to 70% higher mutual information) to the ground truth kinetic parameters (of the simulated abdominal DCE phantom) than the ones derived from the indirect methods. For the prostate DCE data, direct kinetic parameters depict the morphology of the tumour better. To examine the impact on cancer diagnosis, a peripheral zone prostate cancer diagnostic model was employed to calculate a probability map for each method.

Motion correction in Dynamic Contrast Enhanced (DCE-) MRI is challenging because rapid intensity changes can compromise common (intensity based) registration algorithms. In this study we introduce a novel registration technique based on robust principal component analysis (RPCA) to decompose a given time-series into a low rank and a sparse component. This allows robust separation of motion components that can be registered, from intensity variations that are left unchanged. This Robust Data Decomposition Registration (RDDR) is demonstrated on both simulated and a wide range of clinical data. Robustness to different types of motion and breathing choices during acquisition is demonstrated for a variety of imaged organs including liver, small bowel and prostate. The analysis of clinically relevant regions of interest showed both a decrease of error (15-62% reduction following registration) in tissue time-intensity curves and improved areas under the curve (AUC60) at early enhancement.

Attenuation correction is an essential requirement for quantification of positron emission tomography (PET) data. In PET/CT acquisition systems, attenuation maps are derived from computed tomography (CT) images. However, in hybrid PET/MR scanners, magnetic resonance imaging (MRI) images do not directly provide a patient-specific attenuation map. The aim of the proposed work is to improve attenuation correction for PET/MR scanners by generating synthetic CTs and attenuation maps. The synthetic images are generated through a multi-atlas information propagation scheme, locally matching the MRI-derived patient's morphology to a database of MRI/CT pairs, using a local image similarity measure. Results show significant improvements in CT synthesis and PET reconstruction accuracy when compared to a segmentation method using an ultrashort-echo-time MRI sequence and to a simplified atlas-based method.

Dynamic magnetic resonance imaging (MRI) is used in multiple clinical applications, but can still benefit from higher spatial or temporal resolution. A dynamic MR image reconstruction method from partial (k, t)-space measurements is introduced that recovers and inherently separates the information in the dynamic scene. The reconstruction model is based on a low-rank plus sparse decomposition prior, which is related to robust principal component analysis. An algorithm is proposed to solve the convex optimization problem based on an alternating direction method of multipliers. The method is validated with numerical phantom simulations and cardiac MRI data against state of the art dynamic MRI reconstruction methods. Results suggest that using the proposed approach as a means of regularizing the inverse problem remains competitive with state of the art reconstruction techniques. Additionally, the decomposition induced by the reconstruction is shown to help in the context of motion estimation in dynamic contrast enhanced MRI.

A new switching scheme, which combines conventional space vector modulation (SVM) and carrier-based pulse width modulation (CBPWM), is proposed for a three-phase four-leg voltage source inverter (VSI). The proposed algorithm is simpler than conventional three-dimensional space vector modulation (3-D SVM) and easy in implementation in a real-time DSP system, it avoids the selection of a prism and tetrahedron in a 3-D SVM, therefore alleviates the computation burden of the DSP. Also, the concept of near-state pulse width modulation (NSPWM) method, which is used in a three-phase three-leg inverter to reduce the common mode voltage (CMV) noises, can be adopted in this algorithm so as to reduce the common mode voltage for a three-phase four-leg VSI. The feasibility of the proposed modulation technique is verified by both computer simulation and experimental results.

A barrier to the adoption of non-Cartesian parallel magnetic resonance imaging for real-time applications has been the times required for the image reconstructions. These times have exceeded the underlying acquisition time thus preventing real-time display of the acquired images. We present a reconstruction algorithm for commodity graphics hardware (GPUs) to enable real time reconstruction of sensitivity encoded radial imaging (radial SENSE). We demonstrate that a radial profile order based on the golden ratio facilitates reconstruction from an arbitrary number of profiles. This allows the temporal resolution to be adjusted on the fly. A user adaptable regularization term is also included and, particularly for highly undersampled data, used to interactively improve the reconstruction quality. Each reconstruction is fully self-contained from the profile stream, i.e., the required coil sensitivity profiles, sampling density compensation weights, regularization terms, and noise estimates are computed in real-time from the acquisition data itself. The reconstruction implementation is verified using a steady state free precession (SSFP) pulse sequence and quantitatively evaluated. Three applications are demonstrated; real-time imaging with real-time SENSE 1) or k- t SENSE 2) reconstructions, and 3) offline reconstruction with interactive adjustment of reconstruction settings.

This paper describes a quantitative assessment of respiratory motion of the heart and the construction of a model of respiratory motion. Three-dimensional magnetic resonance scans were acquired on eight normal volunteers and ten patients. The volunteers were imaged at multiple positions in the breathing cycle between full exhalation and full inhalation while holding their breath. The exhalation volume was segmented and used as a template to which the other volumes were registered using an intensity-based rigid registration algorithm followed by nonrigid registration. The patients were imaged at inhale and exhale only. The registration results were validated by visual assessment and consistency measurements indicating subvoxel registration accuracy. For all subjects, we assessed the nonrigid motion of the heart at the right coronary artery, right atrium, and left ventricle. We show that the rigid-body motion of the heart is primarily in the craniocaudal direction with smaller displacements in the right-left and anterior-posterior directions; this is in agreement with previous studies. Deformation was greatest for the free wall of the right atrium and the left ventricle; typical deformations were 3-4 mm with deformations of up to 7 mm observed in some subjects. Using the registration results, landmarks on the template surface were mapped to their correct positions through the breathing cycle. Principal component analysis produced a statistical model of the motion and deformation of the heart. We discuss how this model could be used to assist motion correction.

A new method is proposed to reconstruct and analyze the left ventricle (LV) from multiple acoustic window three-dimensional (3-D) ultrasound acquired using a transthoracic 3-D rotational probe. Prior research in this area has been based on one acoustic window acquisition. However, the data suffers from several limitations that degrade the reconstruction and reduce the clinical value of interpretation, such as the presence of shadow due to bone (ribs) and air (in the lungs) and motion of the probe during the acquisition. In this paper, we show how to overcome these limitations by automatically fusing information from multiple acoustic window sparse-view acquisitions and using a position sensor to track the probe in real time. Geometric constraints of the object shape, and spatiotemporal information relating to the image acquisition process, are used in new algorithms for 1) grouping endocardial edge cues from an initial image segmentation and 2) defining a novel reconstruction method that utilizes information from multiple acoustic windows. The new method has been validated on a phantom and three real heart data sets. In the phantom study, one finger of a latex glove was scanned from two acoustic windows and reconstructed using the new method. The volume error was measured to be less than 4%. In the clinical case study, 3-D ultrasound and magnetic resonance imaging (MRI) scanning were performed on the same healthy volunteers. Quantitative ejection fractions (EFs) and volume-time curves over a cardiac cycle were estimated using the new method and compared to cardiac MRI measurements. This showed that the new method agrees better with MRI measurements than the previous approach we have developed based on a single acoustic window. The EF errors of the new method with respect to MRI measurements were less than 6%. A more extensive clinical validation is required to establish whether these promising first results translate to a method suitable for routine clinical use.

We present the use of an entropy focus criterion to enable automatic focusing of motion corrupted magnetic resonance images. We demonstrate the principle using illustrative examples from cooperative volunteers. Our technique can determine unknown patient motion or use knowledge of motion from other measures as a starting estimate. The motion estimate is used to compensate the acquired data and is iteratively refined using the image entropy. Entropy focuses the whole image principally by favoring the removal of motion induced ghosts and blurring from otherwise dark regions of the image. Using only the image data, and no special hardware or pulse sequences, we demonstrate correction for arbitrary rigid-body translational motion in the imaging plane and for a single rotation. Extension to three-dimensional (3-D) and more general motion should be possible. The algorithm is able to determine volunteer motion well. The mean absolute deviation between algorithm and navigator-echo-determined motion is comparable to the displacement step size used in the algorithm. Local deviations from the recorded motion or navigator-determined motion are explained and we indicate how enhanced focus criteria may be derived. In all cases we were able to compensate images for patient motion, reducing blurring and ghosting.