The paper addresses the limitations of traditional evaluation metrics for Question Answering (QA) systems that primarily focus on syntax and n-gram similarity. We propose a novel model-based evaluation metric, MQA-metric, and create a human-judgment-based dataset, squad-qametric and marco-qametric, to validate our approach. The research aims to solve several key problems: the objectivity in dataset labeling, the effectiveness of metrics when there is no syntax similarity, the impact of answer length on metric performance, and the influence of real answer quality on metric results. To tackle these challenges, we designed an interface for dataset labeling and conducted extensive experiments with human reviewers. Our analysis shows that the MQA-metric outperforms traditional metrics like BLEU, ROUGE and METEOR. Unlike existing metrics, MQA-metric leverages semantic comprehension through large language models (LLMs), enabling it to capture contextual nuances and synonymous expressions more effectively. This approach sets a standard for evaluating QA systems by prioritizing semantic accuracy over surface-level similarities. The proposed metric correlates better with human judgment, making it a more reliable tool for evaluating QA systems. Our contributions include the development of a robust evaluation workflow, creation of high-quality datasets, and an extensive comparison with existing evaluation methods. The results indicate that our model-based approach provides a significant improvement in assessing the quality of QA systems, which is crucial for their practical application and trustworthiness.

In the complicated world of legal law, Question Answering (QA) systems only work if they can give correct, situation-aware, and logically sound answers. Traditional evaluation methods, which rely on superficial similarity measures, can’t catch the complex accuracy and reasoning needed in legal answers. This means that evaluation methods need to change completely. To fix the problems with current methods, this study presents a new model-based evaluation metric that is designed to work well with legal QA systems. We are looking into the basic ideas that are needed for this kind of metric, as well as the problems of putting it into practice in the real world, finding the right technological frameworks, creating good evaluation methods. We talk about a theory framework that is based on legal standards and computational linguistics. We also talk about how the metric was created and how it can be used in real life. Our results, which come from thorough tests, show that our suggested measure is better than existing ones. It is more reliable, accurate, and useful for judging legal quality assurance systems.

Abstract Text data have an important place in our daily life. A huge amount of text data is generated everyday. As a result, automation becomes necessary to handle these large text data. Recently, we are witnessing important developments with the adaptation of new approaches in text processing. Attention mechanisms and transformers are emerging as methods with significant potential for text processing. In this study, we introduced a bidirectional transformer (BiTransformer) constructed using two transformer encoder blocks that utilize bidirectional position encoding to take into account the forward and backward position information of text data. We also created models to evaluate the contribution of attention mechanisms to the classification process. Four models, including long short term memory, attention, transformer, and BiTransformer, were used to conduct experiments on a large Turkish text dataset consisting of 30 categories. The effect of using pretrained embedding on models was also investigated. Experimental results show that the classification models using transformer and attention give promising results compared with classical deep learning models. We observed that the BiTransformer we proposed showed superior performance in text classification.

Summary Knapsack is a combinatorial optimization problem that involves a variety of resource allocation challenges. It is defined as non‐deterministic polynomial time (NP) hard and has a wide range of applications. Knapsack problem (KP) has been studied in applied mathematics and computer science for decades. Many algorithms that can be classified as exact or approximate solutions have been proposed. Under the category of exact solutions, algorithms such as branch‐and‐bound and dynamic programming and the approaches obtained by combining these algorithms can be classified. Due to the fact that exact solutions require a long processing time, many approximate methods have been introduced for knapsack solution. In this research, deep Q‐learning using models containing fully connected layers, attention, and transformer as function estimators were used to provide the solution for KP. We observed that deep Q‐networks, which continued their training by observing the reward signals provided by the knapsack environment we developed, optimized the total reward gained over time. The results showed that our approaches give near‐optimum solutions and work about 40 times faster than an exact algorithm using dynamic programming.

Summary Many sources such as historical archives, sensor readings, health systems, and machine records produce ever‐increasing but often unchanging data. These accumulating data create a need for faster processing. Bitmap index, which can take advantage of multi‐core and multiprocessor systems, is designed to process data that increase over time but do not change frequently. It has a well‐known advantage, especially in queries on data with low cardinality. However, bitmap index can handle high cardinality data efficiently because it can use its own compression algorithm. Bitmap index has many encoding schemes that affect query processing time. In this study, we developed an algorithm that improves query performance by using optimal encoding among bitmap encodings. With this optimization algorithm, we witnessed up to 40% performance increase in queries made with bitmap indexes created with different encodings. Furthermore, in comparison with a commonly used relational database, we found significant improvements in the number of query operations per second performed on optimized encoded bitmap indexes generated by the introduced algorithm.

Abstract Deep learning practices have a great impact in many areas. Big data and significant hardware developments are the main reasons behind deep learning success. Recent advances in deep learning have led to significant improvements in text analysis and classification. Progress in the quality of word representation is an important factor among these improvements. In this study, we aimed to develop word2vec word representation, also called embedding, by automatically optimizing hyperparameters. Minimum word count, vector size, window size, negative sample, and iteration number were used to improve word embedding. We introduce two approaches for setting hyperparameters that are faster than grid search and random search. Word embeddings were created using documents of approximately 300 million words. We measured the quality of word embedding using a deep learning classification model on documents of 10 different classes. It was observed that the optimization of the values of hyperparameters alone increased classification success by 9%. In addition, we demonstrate the benefits of our approaches by comparing the semantic and syntactic relations between word embedding using default and optimized hyperparameters.

Summary Many scientific applications produce very large amounts of data as advances in hardware fuel computing and experimental facilities. Managing and analyzing massive quantities of scientific data is challenging as data are often stored in specific formatted files, such as HDF5 and NetCDF, which do not offer appropriate search capabilities. In this research, we investigated a special class of search capability, called membership query, to identify whether queried elements of a set are members of an attribute. Attributes that naturally have classification values appear frequently in scientific domains such as category and object type as well as in daily life such as zip code and occupation. Because classification attribute values are discrete and require random data access, performing a membership query on a large scientific data set creates challenges. We applied bitmap indexing and parallelization to membership queries to overcome these challenges. Bitmap indexing provides high performance not only for low cardinality attributes but also for high cardinality attributes, such as floating‐point variables, electric charge, or momentum in a particle physics data set, due to compression algorithms such as Word‐Aligned Hybrid. We conducted experiments, in a highly parallelized environment, on data obtained from a particle accelerator model and a synthetic data set.

SUMMARY Over the last few decades, distributed systems have architecturally evolved. One recent evolutionary step is SOA. The SOA model is perfectly engendered in Web services, which provide software platforms for building applications as services. Web services utilize supportive capabilities such as security, reliability, and monitoring. These capabilities are typically provisioned as handlers, which incrementally add new features. Even though handlers are very important, the method of utilization is crucial for obtaining potential benefits. Every attempt to support a service with an additional handler increases the chance of an overwhelmingly crowded handler chain. Moreover, a handler may become a bottleneck because of its comparably higher processing time. In this paper, we present the Distributed Handler Architecture to provide an efficient, scalable, and modular architecture. The performance and scalability benchmarks show that the distributed and parallel handler executions are very promising for suitable handler configurations. The paper is concluded with remarks on the fundamentals of a promising computing environment for Web service handlers. Copyright © 2012 John Wiley & Sons, Ltd.

Web service is a standardization effort to interoperate loosely-coupled applications. A Web service interaction benefits and sometimes requires additive functionalities, called as handlers. They contribute to build rich, modular and efficient Web services. However, the way of utilizing them is very crucial for the Web service architecture and its overall performance. Using distributed approach for the handler execution facilitates significantly to obtain full benefit from them. In this paper we describe an orchestration structure for the handlers to attain richer, more modular and efficient Web services.

As Web services have matured they have been substantially leveraged within the academic, research and business communities. An exemplar of this is the realignment, last year, of the dominant grid application framework - Open Grid Services Infrastucture (OGSI) - with the emerging consensus within the Web services community. Reliable messaging is an important component within the Web services stack. There are two competing, and very similar, specifications within this domain viz. WS-ReliableMessaging (WSRM) and WS-reliability (WSR); this work focuses on the WSRM specification. In this paper we provide an overview of the WSRM protocol, describe our implementation of WSRM, and present an analysis of the costs (in terms of latencies and memory utilizations) involved in the use of WSRM. Since WSRM is very similar to WS-reliability we expect the performance of WSRM to be very similar to that of WSR. We hope that the work presented here helps researchers and systems designers gauge the suitability of Web services based reliable messaging in their applications and also to make appropriate trade-offs, which includes inter alia interoperability, guarantees, quality of service and performance.

Abstract In this paper we present the results pertaining to the NaradaBrokering middleware infrastructure. NaradaBrokering is designed to run on a large network of cooperating broker nodes. NaradaBrokering capabilities include, among other things, support for a wide variety of transport protocols, Java Message Service compliance, support for routing JXTA interactions, support for audio/video conferencing applications and, finally, support for multiple constraint specification formats such as XPath, SQL and regular expression queries. This paper demonstrates the suitability of NaradaBrokering to a wide variety of applications and scenarios. Copyright © 2005 John Wiley & Sons, Ltd.