Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers.We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation.We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage.On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes.We hypothesize this is due to high inter-model agreement of answer correctness.To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions.Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning.We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model.Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks.Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point.Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset.Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings.Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

Orith Toledo-Ronen, Matan Orbach, Yoav Katz, Noam Slonim. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Current TSA evaluation in a cross-domain setup is restricted to the small set of review domains available in existing datasets.Such an evaluation is limited, and may not reflect true performance on sites like Amazon or Yelp that host diverse reviews from many domains.To address this gap, we present YASO -a new TSA evaluation dataset of open-domain user reviews.YASO contains 2215 English sentences from dozens of review domains, annotated with target terms and their sentiment.Our analysis verifies the reliability of these annotations, and explores the characteristics of the collected data.Benchmark results using five contemporary TSA systems show there is ample room for improvement on this challenging new dataset.YASO is available at github.com/IBM/yaso-tsa.

Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Roy Bar-Haim, Dalia Krieger, Orith Toledo-Ronen, Lilach Edelstein, Yonatan Bilu, Alon Halfon, Yoav Katz, Amir Menczel, Ranit Aharonov, Noam Slonim. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

One of the challenges that processor level stimuli generators are facing is the need to generate stimuli that exercise microarchitectural mechanisms deep inside the verified processor. These scenarios require specific relations between the instructions participating in them. We present a new approach for processor-level scenario generation. The approach is based on creating an abstract constraint satisfaction problem, which captures the essence of the requested scenario. The generation of stimuli is done by interleaving between progress in the solution of the abstract CSP and generation of instructions. Compared with existing solutions of scenario generation, this approach yields improved coverage and reduced generation fail rate.

Microarchitectural information regarding various aspects of instruction execution can help processor-level stimuli generators more easily reach verification goals. While many such aspects are based on common microarchitectural concepts, their specific manifestations are highly design-specific. We propose using an automatic method for acquiring such microarchitectural knowledge and integrating it into the stimuli generator. We start by extracting microarchitectural data from simulation traces. This data is fed to a decision tree learning algorithm that produces rules for microarchitectural behavior of instructions; these rules are then integrated into the testing knowledge of the stimuli generator. This testing knowledge can provide users with the ability to better control the microarchitectural behavior of generated instructions, leading to higher quality test cases. Experimental results on the POWER7 processor showed that our proposed method can improve the microarchitectural coverage of the design.

We consider the following scheduling with batching problem that has many applications, for example, in multimedia-on-demand and manufacturing of integrated circuits. The input to the problem consists of n jobs and k parallel machines. Each job is associated with a set of time intervals in which it can be scheduled (given either explicitly or nonexplicitly), a weight, and a family. Each family is associated with a processing time. Jobs that belong to the same family can be batched and executed together on the same machine. The processing time of each batch is the processing time of the family of jobs it contains. The goal is to find a nonpreemptive schedule with batching that maximizes the weight of the scheduled jobs. We give constant factor (4 or 4 + ε) approximation algorithms for two variants of the problem, depending on the precise representation of the input. When the batch size is unbounded and each job is associated with a time window in which it can be processed, these approximation ratios reduce to 2 and 2 + ε, respectively. We also give approximation algorithms for two special cases when all release times are the same.

Parallelism in system architecture and design imposes a verification challenge as the exponential growth in the number of execution combinations becomes unwieldy. We report on a method for system-level test case generation. This method relies on dynamic interleaving of scenarios from the core level or sub-system level. We discuss the relevance of this method for the system level. We also describe a tool that implements this method and show how it was used in IBM for system verification of the Xbox 360 chip and power management in the cell processor, as well as verification of the pSeries eServers. We claim that this method shortened the system level verification cycle and allowed reuse in and across projects, which led to exposure of system-level bugs in a relatively short time.

Functional verification of systems is aimed at validating the integration of previously verified components. It deals with complex designs, and invariably suffers from scarce resources. We present a set of methods, collectively known as testing knowledge, aimed at increasing the quality of automatically generated system-level test-cases. Testing knowledge reduces the time and effort required to achieve high coverage of the verified design.