Abstract Transformers have achieved state-of-the-art performance in most common tasks to which they have been applied. Those achievements are attributed to the Self-Attention mechanism at their core. Self-Attention is understood to map the relationship between tokens of any given sequence. This exhaustive mapping incurs massive costs in memory and inference time, as Self-Attention scales quadratically with regard to sequence length. Standard Self-Attention has required increasingly large compute and memory usage when applied to long input sequences because of this memory and time bottleneck. Efficient Transformers emerged as performant alternatives demonstrating good scalability and occasionally better tracking of long-range dependencies. Their efficiency gains are obtained through different methods, usually focusing on the linear scaling of the attention matrix through sparsification, approximation, or other methods. Among existing approaches, those using low-rank approximation present particular advantages because of their compatibility with standard Self-Attention-based models, allowing for weight transfers and other time-saving schemes. More recently, hardware-aware versions of Self-Attention (e.g., FlashAttention) have mitigated all memory bottlenecks and have alleviated its compute burden. Unfortunately, hardware-aware Self-Attentions have stricter hardware compatibility requirements making Efficient Transformers still relevant for use on older or less powerful hardware. Furthermore, some Efficient Transformers can even be applied in an hardware-aware manner to further improve training and inference speed. In this paper, we propose a novel linear approximation method for Self-Attention inspired by the CUR approximation method. This method, proposed in two versions (one leveraging FlashAttention), is conceived as a drop-in replacement for standard Self-Attention with weights compatibility. Our method compares favorably to standard Transformers’ and Efficient Transformers’ performances on varied tasks and demonstrates a significant decrease in memory footprint as well as competitive performance in training speed, even compared to similar methods.

Astrocytes with their specialised morphology are essential for brain homeostasis as metabolic mediators between blood vessels and neurons. In neurodegenerative diseases such as Alzheimer's disease (AD), astrocytes adopt reactive profiles with molecular and morphological changes that could lead to the impairment of their metabolic support and impact disease progression. However, the underlying mechanisms of how the metabolic function of human astrocytes is impaired by their morphological changes in AD are still elusive. To address this challenge, we developed and applied a metabolic multiscale modelling approach integrating the dynamics of metabolic energy pathways and physiological astrocyte morphologies acquired in human AD and age-matched control brain samples. The results demonstrate that the complex cell shape and intracellular organisation of energetic pathways determine the metabolic profile and support capacity of astrocytes in health and AD conditions. Thus, our mechanistic approach indicates the importance of spatial orchestration in metabolism and allows for the identification of protective mechanisms against disease-associated metabolic impairments.

Summary: Ca2+ is a central second messenger in eukaryotic cells that regulates many cellular processes. Recently, we have indicated that typical Ca2+ signals are not purely oscillatory as widely assumed, but exhibit stochastic spiking with cell type and pathway specific characteristics. Here, we present the Calcium Signaling Analyzer (CaSiAn), an open source software tool that allows for quantifying these signal characteristics including individual spike properties and time course statistics in a semi-automated manner. CaSiAn provides an intuitive graphical user interface allowing experimentalists to easily process a large amount of Ca2+ signals, interactively tune peak detection, revise statistical measures and access the quantified signal properties as excel or text files. Availability and implementation: CaSiAn is implemented in Java and available on Github (https://github.com/mmahsa/CaSiAn) as well as on the project page (http://r3lab.uni.lu/web/casa). Supplementary information: Supplementary data are available at Bioinformatics online.

Usually, the occurrence of random cell behavior is appointed to small copy numbers of molecules involved in the stochastic process. Recently, we demonstrated for a variety of cell types that intracellular Ca2+ oscillations are sequences of random spikes despite the involvement of many molecules in spike generation. This randomness arises from the stochastic state transitions of individual Ca2+ release channels and does not average out due to the existence of steep concentration gradients. The system is hierarchical due to the structural levels channel--channel cluster--cell and a corresponding strength of coupling. Concentration gradients introduce microdomains which couple channels of a cluster strongly. But they couple clusters only weakly; too weak to establish deterministic behavior on cell level. Here, we present a multi-scale modelling concept for stochastic hierarchical systems. It simulates active molecules individually as Markov chains and their coupling by deterministic diffusion. Thus, we are able to follow the consequences of random single molecule state changes up to the signal on cell level. To demonstrate the potential of the method, we simulate a variety of experiments. Comparisons of simulated and experimental data of spontaneous oscillations in astrocytes emphasize the role of spatial concentration gradients in Ca2+ signalling. Analysis of extensive simulations indicates that frequency encoding described by the relation between average and standard deviation of interspike intervals is surprisingly robust. This robustness is a property of the random spiking mechanism and not a result of control.