Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations poses a challenge, leading to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to condense lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LLM's in-context learning performance. To mitigate these challenges, we present Meta Demonstration Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. We exploit the knowledge distillation to enhance alignment between MEND and MEND, achieving both efficiency and effectiveness concurrently. MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning. Comprehensive evaluations across seven diverse ICL settings using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess. It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. This innovation promises enhanced scalability and efficiency for the practical deployment of large language models.

Recent advances in cross-lingual commonsense reasoning (CSR) are facilitated by the development of multilingual pre-trained models (mPTMs). While mPTMs show the potential to encode commonsense knowledge for different languages, transferring commonsense knowledge learned in large-scale English corpus to other languages is challenging. To address this problem, we propose the attention-based Cross-LIngual Commonsense Knowledge transfER (CLICKER) framework, which minimizes the performance gaps between English and non-English languages in commonsense question-answering tasks. CLICKER effectively improves commonsense reasoning for non-English languages by differentiating non-commonsense knowledge from commonsense knowledge. Experimental results on public benchmarks demonstrate that CLICKER achieves remarkable improvements in the cross-lingual CSR task for languages other than English.

The Transformer Memory as a Differentiable Search Index (DSI) has been proposed as a new information retrieval paradigm, which aims to address the limitations of dual-encoder retrieval framework based on the similarity score. The DSI framework outperforms strong baselines by directly generating relevant document identifiers from queries without relying on an explicit index. The memorization power of DSI framework makes it suitable for personalized retrieval tasks. Therefore, we propose a Personal Transformer Memory (PersonalTM) architecture for personalized text retrieval. PersonalTM incorporates user-specific profiles and contextual user click behaviors, and introduces hierarchical loss in the decoding process to align with the hierarchical assignment of document identifier. Additionally, PersonalTM also employs an adapter architecture to improve the scalability for index updates and reduce computation costs, compared to the vanilla DSI. Experiments show that PersonalTM outperforms the DSI baseline, BM25, fine-tuned dual-encoder, and other personalized models in terms of precision at top 1st and 10th positions and Mean Reciprocal Rank (MRR). Specifically, PersonalTM improves p@1 by 58%, 49%, and 12% compared to BM25, Dual-encoder, and DSI, respectively.

With billions of networked connected embedded systems, the security historically provided by the isolation of embedded systems is no longer sufficient. Millions of new malware are created every month and zero-day attacks are becoming an increasing concern. Therefore, proactive security measures are no longer enough to provide protection to embedded systems. Instead, reactive approaches that detect attacks that can circumvent the proactive defenses and react upon them are needed. Anomaly-based detection is a common reactive approach employed to detect malware by monitoring anomalous deviations in the system execution. Timing-based anomaly detection detects malware by monitoring the system's internal timing, which offers unique protection against mimicry malware compared to sequence-based anomaly detection. However, previous timing-based anomaly detection methods focus on each operation independently at the granularity of tasks, function calls, system calls, or basic blocks. These approaches neither consider the entire software execution path nor provide a quantitative estimate of the presence of malware. This article presents a novel model for specifying the normal timing for execution paths in software applications using cumulative distribution functions of timing data in sliding execution windows. A probabilistic formulation is used to estimate the presence of malware for individual operations and sequences of operations within the paths. Operation and path-based thresholds are determined during the training process to minimize false positives. Finally, the article presents an optimization method to assist system developers in selecting which operations to monitor based on different optimization goals and constraints. Experimental results with a smart connected pacemaker, an unmanned aerial vehicle, and seven sophisticated mimicry malware implemented at different levels demonstrate the effectiveness of the proposed approach.

Malware is a serious threat to network-connected embedded systems, as evidenced by the continued and rapid growth of such devices, commonly referred to as the Internet of Things. Their ubiquitous use in critical applications require robust protection to ensure user safety and privacy. That protection must be applied to all system aspects, extending beyond protecting the network and external interfaces. Anomaly detection is one of the last lines of defence against malware, in which data-driven approaches that require the least domain knowledge are popular. However, embedded systems, particularly edge devices, face several challenges in applying data-driven anomaly detection, including unpredictability of malware, limited tolerance to long data collection windows, and limited computing/energy resources. In this article, we utilize subcomponent timing information of software execution, including intrinsic software execution, instruction cache misses, and data cache misses as features, to detect anomalies based on ranges, multi-dimensional Euclidean distance, and classification at runtime. Detection methods based on lumped timing range are also evaluated and compared. We design several hardware detectors implementing these data-driven detection methods, which non-intrusively measuring lumped/subcomponent timing of all system/function calls of the embedded application. We evaluate the area, power, and detection latency of the presented detector designs. Experimental results demonstrate that the subcomponent timing model provides sufficient features to achieve high detection accuracy with low false-positive rates using a one-class support vector machine, considering sophisticated mimicry malware.

With billions of networked connected embedded systems, the security historically provided by the isolation of embedded systems is no longer sufficient. Both proactive security measures that prevent intrusions and reactive measures that detect intrusions are essential. Anomaly-based detection is a common reactive approach employed to detect malware that has evaded proactive defenses by observing anomalous deviations in the system execution. Timing-based anomaly detection detects malware by monitoring the system's internal timing, which offers unique protection against mimicry malware compared to sequence-based anomaly detection. However, previous timing-based anomaly detection methods focus on each operation independently at the granularity of tasks, function calls, system calls, or basic blocks. These approaches neither consider the entire software execution path nor provide a quantitative estimate of the presence of malware. This paper presents a novel model for specifying the normal timing for execution paths in software applications using cumulative distribution functions of timing data in sliding execution windows. We present a probabilistic formulation for estimating the presence of malware for individual operations and sequences of operations within the paths, and we define thresholds to minimize false positives based on training data. Experimental results with a smart connected pacemaker and three sophisticated mimicry malware demonstrate improved performance and accuracy compared to state-of-the-art timing-based malware detection.

Network-connected embedded systems grow on a large scale as a critical part of Internet of Things, and these systems are under the risk of increasing malware. Anomaly-based detection methods can detect malware in embedded systems effectively and provide the advantage of detecting zero-day exploits relative to signature-based detection methods, but existing approaches incur significant performance overheads and are susceptible to mimicry attacks. In this article, we present a formal runtime security model that defines the normal system behavior including execution sequence and execution timing. The anomaly detection method in this article utilizes on-chip hardware to non-intrusively monitor system execution through trace port of the processor and detect malicious activity at runtime. We further analyze the properties of the timing distribution for control flow events, and select subset of monitoring targets by three selection metrics to meet hardware constraint. The designed detection method is evaluated by a network-connected pacemaker benchmark prototyped in FPGA and simulated in SystemC, with several mimicry attacks implemented at different levels. The resulting detection rate and false positive rate considering constraints on the number of monitored events supported in the on-chip hardware demonstrate good performance of our approach.

Network-connected embedded systems require multiple lines of defense against malware. In addition to preventing malware by designing secure interfaces and software, anomaly-based detection is needed to detect malware that successfully infiltrates these defenses. Timing based anomaly detection strengthens embedded system security by detecting anomalies in the execution time of critical software tasks. However, existing timing based anomaly detection methods use a lumped timing model that aggregates the timing of the software, processor architecture, operating system scheduling, etc., and thereby incurs significant variability. We present a non-intrusive hardware detector supporting two novel timing models, including a lumped timing multi-range model that clusters timing into multiple range bounds, and a subcomponent timing model that defines bounds for timing subcomponents of events. Timing subcomponents include intrinsic software execution, instruction cache misses, data cache misses, and interrupts. The experimental results demonstrate that the detection based on subcomponent timing model achieves greater malware detection accuracy compared to the lumped timing model without increasing false positives.

Recent research has demonstrated that many systems are vulnerable to numerous types of malicious activity. As the pervasiveness of embedded systems with network connectivity continues to increase, embedded systems security has become a critical challenge. However, most existing techniques for detecting malware utilize software-based methods that incur significant performance overheads that are often not feasible in embedded systems. In this paper, we present an overview of a novel method for non-intrusively detecting malware in embedded system. The proposed technique utilizes timing requirements to improve detection performance and provide increased resilience to mimicry attacks.

Joint User and Antenna Selection (JUAS) with global optimal solution for multi-cell Multi-User Multiple Input Multiple Output (MU-MIMO) systems needs the centralized processing and full Channel State Information (CSI) sharing that both require high capacity backhaul links. In this paper, decentralized JUAS algorithm is considered for multi-cell MU-MIMO systems with Block Diagonalization (BD) precoding. First, by introducing the eigenvalue-based lower bound on the capacity, the global optimal problem of JUAS is reformulated to a new problem that can achieve local optimum. Second, a decentralized fairness-based algorithm is proposed to select the antenna subset with no CSI sharing. Simulation results demonstrate that the proposed algorithm outperforms existing algorithms, such as capacity-based antenna selection and interference-aware user selection, with the same computation complexity. Moreover, the performance losses compared to centralized JUAS with full CSI sharing are marginal.

Uplink Inter-Cell Interference (ICI) and interference fluctuation degrades the uplink throughput of LTE/LTE-Advanced systems seriously. To deal with these problems, a coordinated scheduling method with resource sorting is proposed. In the proposed method, the uplink interference fluctuation is resisted by exploiting resource sorting and limiting the ICI generated by a given user to a predefined range. Meanwhile, the resources are allocated by differentiating the users' ability to bear the interference. Simulation results show that the proposed method outperforms traditional ICIC schemes, such as Fractional Frequency Reuse, fractional Transmit Power Control (TPC), and overload indicator based TPC.