Large Language Models for code often entail significant computational complexity, which grows significantly with the length of the input code sequence.We propose LEANCODE for code simplification to reduce training and prediction time, leveraging code contexts in utilizing attention scores to represent the tokens' importance.We advocate for the selective removal of tokens based on the average context-aware attention scores rather than average scores across all inputs.LEANCODE uses the attention scores of 'CLS' tokens within the encoder for classification tasks, such as code search.It also employs the encoderdecoder attention scores to determine token significance for sequence-to-sequence tasks like code summarization.Our evaluation shows LEANCODE's superiority over the SOTAs DI-ETCODE and SLIMCODE, with improvements of 60% and 16% for code search, and 29% and 27% for code summarization, respectively.

Long-context language models (LCLMs) have exhibited impressive capabilities in longcontext understanding tasks.Among these, long-context referencing-a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data-remains underexplored.To bridge this gap, this paper proposes Referencing Evaluation for Longcontext Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs.Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval.Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark.Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o.To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights.Our data and code can be found in https://github. com

Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, *how well do the models truly perform visual temporal reasoning?* Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) *Multi-Frame Gain*, (2) *Frame Order Sensitivity*, and (3) *Frame Information Disparity*. Following these principles, we introduce **TOMATO**, **T**emp**O**ral Reasoning **M**ultimod**A**l Evalua**T**i**O**n, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, *human-annotated* questions spanning *six* tasks (i.e. *action count, direction, rotation, shape & trend, velocity & frequency, and visual cues*), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model. Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence. We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending the human world dynamics through the video modality.

Generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization.In this work, we challenge this convention by showing that discriminative approaches perform substantially better than generative ones on a large number of NLP tasks.Technically, we train a single discriminator to predict whether a text sample comes from the true data distribution, similar to GANs.Since many NLP tasks can be formulated as selecting from a few options, we use this discriminator to predict the concatenation of input and which option has the highest probability of coming from the true data distribution.This simple formulation achieves state-of-theart zero-shot results on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales.In the finetuning setting, our approach also achieves new stateof-the-art results on a wide range of NLP tasks, with only 1/4 parameters of previous methods.Meanwhile, our approach requires minimal prompting efforts, which largely improves robustness and is essential for real-world applications.Furthermore, we also jointly train a generalized UD in combination with generative tasks, which maintains its advantage on discriminative tasks and simultaneously works on generative tasks.

Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Chonghua Liao, Li Jian, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Most previous methods for text data augmentation are limited to simple tasks and weak baselines.We explore data augmentation on hard tasks (i.e., few-shot natural language understanding) and strong baselines (i.e., pretrained models with over one billion parameters).Under this setting, we reproduced a large number of previous augmentation methods and found that these methods bring marginal gains at best and sometimes degrade the performance much.To address this challenge, we propose a novel data augmentation method FlipDA that jointly uses a generative model and a classifier to generate label-flipped data.Central to the idea of FlipDA is the discovery that generating labelflipped data is more crucial to the performance than generating label-preserved data.Experiments show that FlipDA achieves a good tradeoff between effectiveness and robustness-it substantially improves many tasks while not negatively affecting the others. 1

Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient learning framework, TLM, that does not rely on large-scale pretraining. Given some labeled task data and a large general corpus, TLM uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch. On eight classification datasets in four domains, TLM achieves results better than or similar to pretrained language models (e.g., RoBERTa-Large) while reducing the training FLOPs by two orders of magnitude. With high accuracy and efficiency, we hope TLM will contribute to democratizing NLP and expediting its development.

With the rapid development of mobile devices, limited licensed spectrum resources can no longer meet the explosive growth of data traffic demand. Therefore, the industry and academia have set their sights on utilizing the unlicensed spectrum in the cellular network which is mainly used by the Wi-Fi network. As a result, it is crucial for the cellular network to harmoniously coexist with the Wi-Fi networks. Recently, the new radio in unlicensed spectrum (NR-U) network has been proposed to address the coexistence between the cellular and Wi-Fi networks from the perspective of beam or space. However, the spatial resource in cellular base stations (BSs), especially in the small-cell BSs, may be insufficient. In this work, we propose a novel joint spatial-temporal domain based scheme to address this. The joint resource allocation problem is formulated to maximize the total throughput of the coexisting NR-U and Wi-Fi networks. By using the mixed integer quadratic programming (MIQP), a joint spatial-temporal resource allocation scheme is obtained. Simulation results show that the joint spatial-temporal domain coexistence scheme can achieve a maximum of 30% and 58.3% gain in terms of the total throughput performance as compared with the sole carrier sense adaptive transmission (CSAT) and interference nulling scheme, respectively, under insufficient spatial resources.

Uplink transmission delay has been considered as a main component of the end-to-end delay in the fog radio access networks (F-RAN). However, existing analysis of the transmission delay ignore the packet retransmission delay, i.e., mean local delay (MLD), which is the main component of transmission delay. In addition, the MLD has not been investigated in a clustered F-RAN. Therefore, in this work, we leverage the Matern cluster process (MCP) to analyse MLD in a large-scale F-RAN. To derive the MLD, we obtain the uplink coverage probability (CP) firstly. The MLD can be derived by the moment result of the CP, which is defined as the probability of received signal-to-interference-ratio (SIR) is larger than a threshold. To obtain the moment result of the CP, the analytical result of the CP and its approximation with a lower computational complexity are derived. However, the moment result of CP is difficult to be validated via simulations. Therefore, we derive the meta distribution, which is calculated directly from the moment result of the CP, and validate its correctness by Monte Carlo simulations. Equipped with this, the analytical results of MLD in closed-form are derived. Based on these results, the effect of UE activity factor and the uplink power control (PC) factor on the MLD are analysed. The results show that the fog access points (FAP) density has no effect on the MLD.

The variational neural models have achieved significant progress in dialogue generation. They are of encoder-decoder architecture, with stochastic latent variables learned at the utterance level. However, latent variables are usually approximated by factorized-form distributions, the value space of which is too large relative to latent features to be encoded, leading to the sparsity problem. As a result, little useful information is carried in latent representations, and generated responses tend to be non-committal and meaningless. To address it, we initially propose the Latent-Constrained Variational Neural Dialogue Model (LC-VNDM). It follows variational neural dialogue framework, with an utterance encoder, a context encoder and a response decoder hierarchically organized. Particularly, LC-VNDM uses a hierarchically-structured variational distribution form, which considers inter-dependencies between latent variables. Thus it defines a constrained latent value space, and prevents latent global features from being diluted. Therefore, latent representations sampled from it would carry richer global information to facilitate the decoding, generating meaningful responses. We conduct extensive experiments on three datasets using automatic evaluation and human evaluation. Experiments prove that LC-VNDM significantly outperforms the state-of-the-arts and can generate information-richer responses by learning a better-quality latent space.

Deep Generative Models (DGMs) are able to extract high-level representations from massive unlabeled data and are explainable from a probabilistic perspective. Such characteristics favor sequence modeling tasks. However, it still remains a huge challenge to model sequences with DGMs. Unlike real-valued data that can be directly fed into models, sequence data consist of discrete elements and require being transformed into certain representations first. This leads to the following two challenges. First, high-level features are sensitive to small variations of inputs as well as the way of representing data. Second, the models are more likely to lose long-term information during multiple transformations. In this paper, we propose a Hierarchical Deep Generative Model With Dual Memory to address the two challenges. Furthermore, we provide a method to efficiently perform inference and learning on the model. The proposed model extends basic DGMs with an improved hierarchically organized multi-layer architecture. Besides, our model incorporates memories along dual directions, respectively denoted as broad memory and deep memory. The model is trained end-to-end by optimizing a variational lower bound on data log-likelihood using the improved stochastic variational method. We perform experiments on several tasks with various datasets and obtain excellent results. The results of language modeling show our method significantly outperforms state-of-the-art results in terms of generative performance. Extended experiments including document modeling and sentiment analysis, prove the high-effectiveness of dual memory mechanism and latent representations. Text random generation provides a straightforward perception for advantages of our model.

BACKGROUND: It has long been recognized that sensitivity analysis plays a key role in modeling and analyzing cellular and biochemical processes. Systems biology markup language (SBML) has become a well-known platform for coding and sharing mathematical models of such processes. However, current SBML compatible software tools are limited in their ability to perform global sensitivity analyses of these models. RESULTS: This work introduces a freely downloadable, software package, SBML-SAT, which implements algorithms for simulation, steady state analysis, robustness analysis and local and global sensitivity analysis for SBML models. This software tool extends current capabilities through its execution of global sensitivity analyses using multi-parametric sensitivity analysis, partial rank correlation coefficient, SOBOL's method, and weighted average of local sensitivity analyses in addition to its ability to handle systems with discontinuous events and intuitive graphical user interface. CONCLUSION: SBML-SAT provides the community of systems biologists a new tool for the analysis of their SBML models of biochemical and cellular processes.