Retrieval in Retrieval-Augmented Generation (RAG) must ensure that retrieved passages are not only individually relevant but also collectively form a comprehensive set.Existing approaches primarily rerank top-k passages based on their individual relevance, often failing to meet the information needs of complex queries in multi-hop question answering.In this work, we propose a set-wise passage selection approach and introduce SETR, which explicitly identifies the information requirements of a query through Chain-of-Thought reasoning and selects an optimal set of passages that collectively satisfy those requirements.Experiments on multi-hop RAG benchmarks show that SETR outperforms both proprietary LLM-based rerankers and open-source baselines in terms of answer correctness and retrieval quality, providing an effective and efficient alternative to traditional rerankers in RAG systems.

Blockchain scalability is an active area of research. One of the promising approaches is the execute-order-validate (X-O-V) blockchain architecture, which leverages high concurrency. Meanwhile, many transactions exhibit dynamic locality—a workload characteristic with a high degree of locality that changes over time (e.g., due to the mobility of devices). In this article, we propose DyloChain, a hierarchical blockchain supporting dynamic locality by extending the X-O-V architecture across sharding zones. In DyloChain’s hierarchy, higher-level servers (called M-nodes) are responsible for cross-zone communication with a main chain to ensure global consistency and availability, while lower-level servers (called L-nodes) validate local blocks with a local chain. To support dynamic locality, DyloChain suggests an order-execute-order-validate (O-X-O-V) transaction processing model that extends the X-O-V model to accommodate the hierarchical structure. Specifically, the O-X-O-V model optimizes efficiency through speculative local updates on each local chain (O-X) for the early removal of potential conflicts within a zone and enables high parallelism across zones. Subsequently, the model aggregates these local blocks onto the main chain (O-V), ensuring a globally consistent total order across zones. This is facilitated by an extended validation procedure that detects interferences between concurrently dependent transactions across zones. DyloChain resolves interferences using a state synchronization protocol that synchronizes the corresponding main states of the interfered local states into the involved local chains. DyloChain improves locality by employing a state reshard protocol that transfers a user’s state ownership between source and destination local chains by atomically committing user-issued reshard requests to the involved chains. We implement and conduct comprehensive performance evaluations of DyloChain under various configurations.

Proof-of-Authority (PoA) is one of the popular blockchain consensus protocols, particularly for permissioned blockchains. However, PoA (as in Clique) still faces performance scalability issues. Its leader-based design restricts throughput and results in resource inefficiency, as it permits only one block to be accepted in each round, discarding all other concurrently proposed blocks in the same round. In this paper, we introduce DaPoA, an effort to enhance the Ethereum PoA Clique network to improve its performance scalability. The main idea of DaPoA is that it follows a parallel leader architecture that allows multiple sealers to propose their blocks in each round, which are then accepted in the same round. To achieve this, DaPoA employs a state-of-the-art directed acyclic graph (DAG)-based Byzantine Fault Tolerant (BFT) consensus algorithm, which enables parallel block proposals for high scalability with leader-less design. We have implemented and evaluated DaPoA on top of the Ethereum PoA Clique to demonstrate its performance scalability. In our setting of eight nodes, DaPoA achieves a 2.47 x higher throughput and 5.76x lower latency compared to Clique.

Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations.

Seongyun Lee, Sue Hyun Park, Yongrae Jo, Minjoon Seo. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Many Blockchain-as-a-Service (BaaS) providers have emerged with the growing interest in BaaS among enterprises. However, current BaaS providers can pose a potential security threat in the context of a centralized service provider and for clients that depend on the provider. In this study, we first consider the problem of auditing BaaS and develop an Enforcer architecture for trustworthy BaaS.