Comprehensive evaluation of large language models (LLMs) typically requires large-scale benchmarks, which is costly in terms of both data annotation and computational resource needed for evaluation.To mitigate these challenges, we propose an efficient evaluation framework that selects a question subset based on pre-tested results, thereby reducing the costs.We formulate the subset selection problem as an optimization task, solved using optimal random sampling and simulated annealing algorithms.We compare our approach with prior clustering-based methods and assess their reliability in terms of score accuracy.Additionally, we perform semantic analysis and evaluate whether the selected subsets preserve the semantic information of the original benchmark using Wasserstein distance.Experimental results show that our method outperforms previous approaches in terms of reliability, as measured by L2 norm.Our study provides an optimized perspective for balancing evaluation efficiency and reliability in LLM assessments, while revealing the relationship between optimization methods and semantic retention.

OBJECTIVE: This study aimed to develop and validate computerized neuropsychological assessment devices for screening patients with mild cognitive impairment (MCI). METHODS: We conducted this study in three phases. Phase I involved the development of a conceptual framework of Memory Guard (MG) based on the principles of the cognitive design system (CDS). Phase II involved three steps of feature engineering: item development, filter, and wrapper. Based on the initial items, the number of items in each dimension was determined through analytic hierarchy process. We constructed an initial set with a total of 198 items with three levels of difficulty. Next, we performed feature selection through comprehensive reliability and validity tests, which resulted in the best item bank of 38 test items. The features for modeling were obtained from the best item bank (option scores, reading time scores and total time scores), demographic variables and their MoCA groups. Regarding the heterogeneity of the feature space, we combined the AdaBoost with the Naive Bayes classification algorithm as the decision model of MG. For the screening tool to be used repeatedly, the retrieval practice effect was considered in the design. Phase III involved the validation of measuring instruments. The features incorporated into the modeling process were optimized based on the classification accuracy and area under curve. We also verified the classification effect of the other three classification models with MG. RESULTS: After three steps of feature engineering, a total of 6 dimensions of cognitive areas were included in MG: orientation, memory, attention, calculation, recall, and language & executive function. 38 features were included in the model (17 features of option score, 20 features of time score, and 1 demographic feature). A total of 333 individuals from two communities in Shanghai and Henan province were included in the measuring instrument verification process. Women accounted for 68.2% of the sample. The median age was 63. 15.3% of the participants had bachelor's degrees or above and 111 participants lived in urban areas (33.3%). The results showed that MG had an accuracy of 93.75% and AUC of 0.923, with a sensitivity of 91.67% and a specificity of 95.45%. Compared to the other three classification models, MG that combined the AdaBoost with the Naive Bayes classification algorithm was the most accurate classifier. CONCLUSIONS: MG was proved to be reliable and valid in early screening for patients with MCI. MG that integrated heterogeneous features such as demography, option scores, and time scores had a better predictive performance for screening MCI.

With the number of cores and working sets of parallel workloads soaring, shared L2 caches exhibit fewer misses than private L2 caches via making better use of the all available cache capacity. However, shared L2 caches induce higher overall L1 miss latencies because of longer average distance between requestor and home node, and potentially congestions at some nodes. We observe that there is a high probability that the requested data of an L1 miss resides in a neighbor node's L1 cache. In such cases, these long-distance accesses to the home nodes can be potentially avoided. In order to successfully leverage the aforementioned property, we propose Bayesian theory oriented Optimal Data-Provider Selection (ODPS). ODPS partitions the multi-core into clusters of 2×2 nodes, and introduces the Proximity Data Prober (PDP) to detect whether an L1 miss can be served by one L1 cache within the same cluster. Furthermore, we devise the Bayesian Decision Classifier (BDC) to intelligently and adaptively select a remote L2 cache or a neighboring L1 node as the data provider according to the minimal miss cost based on the Bayesian decision theory.

H.264 video codec system requires big capacity of Frame Store (FS) for buffering reference frames. The up-to-date Phase-change Random Access Memory (PRAM) is the promising approach for on-chip caching the reference signals, as PRAM offers the advantages in terms of high density and low leakage power. However, the write endurance problem, that is a PRAM cell can only tolerant limited number of write operations, becomes the main barrier in practical applications. This paper studies the wear reduction techniques of PRAM based FS in H.264 codec system. On the basis of rate-distortion theory, the content oriented selective writing mechanisms are proposed to reduce bit updates in the reference frame buffers. Experiments demonstrate that, for typical video sequences with different frame sizes, our methods averagely achieve more than 30% reduction of bit updates, while introducing around 20% BDBR cost. The power consumption is reduced by 55% on average, and the estimated PRAM lifetime is extended by 61%.

As the number of cores and the working sets of parallel workloads increase, shared L2 caches exhibit fewer misses than private L2 caches by making a better use of the total available cache capacity, but they also induce higher overall L1 miss latencies because of the longer average distance between two nodes, and the potential congestions at certain nodes. One of the main causes of the long L1 miss latencies are accesses to home nodes of the directory. However, we have observed that there is a high probability that the target data of an L1 miss resides in the L1 cache of a neighbor node. In such cases, these long-distance accesses to the home nodes can be potentially avoided. We organize the multi-core into clusters of 2×2 nodes, and in order to leverage the aforementioned property, we introduce the Cluster Cache Monitor (CCM). The CCM is a hardware structure in charge of detecting whether an L1 miss can be served by one of the cluster L1 caches, and two cluster-related states in the coherence protocol in order to avoid long-distance accesses to home nodes upon hits in the cluster L1 caches. We evaluate this approach on a 64-node multi-core using SPLASH-2 and PARSEC benchmarks, and we find that the CCM can reduce the execution time by 15% and reduce the energy by 14%, while saving 28% of the directory storage area compared to a standard multi-core with a shared L2. We also show that the CCM outperforms recent mechanisms, such as ASR, DCC and RNUCA.

Phase-change Random Access Memory (PRAM) is one of the most promising technologies among emerging non-volatile memory technologies, which provides many benefits, such as high density, non-volatility and low leakage power. However, the limited write endurance of PRAM prevents it from being used as a drop-in replacement of SRAM cache. Moreover, the inherent high latency and power dissipation of write operations are both hindrances that PRAM faces. In this paper, we study the L2 cache write operations incurred by different types of data, and accordingly, propose Wear-Resistant Hybrid Cache Architecture (WRHCA), in which the write access behavior of the hybrid L2 cache, that is composed of SRAM and PRAM, is optimized. Through the prediction of data access patterns, the proposed WRHCA prevents write-prone data from entering PRAM L2 cache, and consequently, the wear-out issue of PRAM is alleviated efficiently. Experimental results on the basis of the trace-driven simulator demonstrate that, as compared to the baseline system with pure PRAM L2 cache, our optimized WRHCA saved 85.5% write operations to PRAM on average, and boosted the performance by the averaged 6.4% CPI reduction. Last but not least, as compared with the primitive 3-level SRAM cache with the same chip area, our WRHCA achieved 60.9% reduction in terms of power consumption.

A high performance digital demodulator for RFID reader receiver is proposed. After carefully analyzing the characteristics of the received data, correlation operation is introduced into the demodulator to estimate the frequency of the received data and decode them. Simulation result indicates that this demodulator can work well under the 6dB input signal-to-noise ratio (SNR) with +/- 22% frequency deviation tolerance. FPGA implementation result shows that the communication distance between reader and tag can be extended to 5m above by the use of this demodulator.