Numerous recent works target to extend effective context length for language models and various methods, tasks and benchmarks exist to measure model's effective memorization length.However, through thorough investigations, we find limitations for currently existing evaluations on model's memorization capability.We provide an extensive survey for limitations in this work and propose a new method called forgetting curve to measure the memorization capability of long-context models.We show that forgetting curve has the advantage of being robust to the tested corpus and the experimental settings, of not relying on prompts and can be applied to any model size.We apply our forgetting curve to a large variety of models involving both transformer and RNN/SSM based architectures.Our measurement provides empirical evidence for the effectiveness of transformer extension techniques while raises questions for the effective length of RNN/SSM based models.We also examine the difference between our measurement and existing benchmarks as well as popular metrics for various models.Our code and results can be found at https://github.com/1azybug/ForgettingCurve.

In this paper, we present a two-layer precoding approach for integrated sensing and communication (ISAC) multi-input multi-output (MIMO) systems. The proposed approach comprises an outer layer precoder that leverages channel state information for beam scheduling and reduces the dimensionality of the channel matrix, thereby improving the achievable sum rate. In addition, the inner layer precoder incorporates a beamforming approach for the sensing function, integrating MIMO radar and multi-user MIMO communication systems. This approach helps to satisfy the signal-to-interference-plus-noise ratio requirements for communication and utilizes the minimum mean square error criterion to approximate the beam pattern of the MIMO radar, achieving a superior sensing performance. Our research demonstrates the capability of improving the performance of ISAC MIMO systems.

Seq2seq models based on Recurrent Neural Networks (RNNs) have recently received a lot of attention in the domain of Semantic Parsing. While in principle they can be trained directly on pairs (natural language utterances, logical forms), their performance is limited by the amount of available data. To alleviate this problem, we propose to exploit various sources of prior knowledge: the well-formedness of the logical forms is modeled by a weighted context-free grammar; the likelihood that certain entities present in the input utterance are also present in the logical form is modeled by weighted finite-state automata. The grammar and automata are combined together through an efficient intersection algorithm to form a soft guide (“background”) to the RNN.We test our method on an extension of the Overnight dataset and show that it not only strongly improves over an RNN baseline, but also outperforms non-RNN models based on rich sets of hand-crafted features.

International audience

We restate the classical logical notion of generation/parsing reversibility in terms of feasible probabilistic sampling, and argue for an implementation based on finite-state factors.We propose a modular decomposition that reconciles generation accuracy with parsing robustness and allows the introduction of dynamic contextual factors.(Opinion Piece)