Modern 5G networks offer a network-sliced infrastructure where each network slice contains a dedicated 5G core software service layer. The 5G core software services in each slice shares common core network resources to meet specific customer needs. A primary challenge in 5G network slicing involves resource sharing and efficient network slice orchestration. Container-based methodologies, including tools like Docker and Kubernetes, have become popular for orchestrating 5G network slice services and managing configurations in microservices-based cloud-native service deployment. However, despite their utility, these tools present significant challenges. Their complexity often necessitates dedicated DevOps teams for effective management, while configuration management can prove arduous, and end-to-end supply chain oversight is lacking. To address these challenges, this paper introduces “Llama-Recipe,” a cloud-native 5G-core service deployment and orchestration platform integrating Generative AI, SBOM, PBOM and NFT. 5G-core service configurations across different network slices are represented as “HOCON (Human-Optimized Config Object Notation)” config objects adhering to the GitOps paradigm. Leveraging custom-trained Meta's Llama2 LLM, Llama-Recipe generates the Kubernetes manifests for network-sliced 5G-core services based on the defined HOCON configurations. The generated Kubernetes manifests of the 5G-core services are deployed in designated Kubernetes clusters utilizing GitOps tools (e.g., ArgoCD), ensuring seamless and automated deployment processes. Additionally, Llama-Recipe introduced a novel mechanism to handle end-to-end supply chain verification of 5G-core software services using Software-Bill of Materials (SBOM) and Pipeline-Bill of Materials (PBOM). SBOMs track all the dependencies and PBOMs facilitate the comprehensive tracking of end-to-end supply chain data for 5G-core software services, enhancing transparency and security. These PBOMs are also generated using the fine-tuned Meta's Llama-2 LLM and are encoded as NFT tokens with a novel NFT token schema. This schema enables easy verification and validation of supply-chain data during deployments, thus helping to prevent various supply-chain attacks. To fine-tune the Meta's Llama2 LLM, we've undertaken a meticulous training process, collaborating with Qlora to transform a 4-bit quantized pre-trained language model into Low-Rank Adapters(LoRA). The effectiveness of the Llama-Recipe is demonstrated through a real-world test-bed deployment in a sliced network scenario, utilizing multiple 5G cores (i.e., Open5GS) across Ericsson's new Radio Access Network (RAN).

Current wind energy data platforms face significant challenges in securing and managing extensive data from both offshore and onshore wind farms. These challenges include vulnerabilities to cyber-attacks, data tampering, breaches, complex data-sharing issues due to privacy concerns and regulatory compliance, and a lack of scalability and flexibility in analytical tools for real-time data processing. This paper proposes a novel multilayered data security architecture, termed "VindSec-Llama," to address these challenges. It integrates Generative AI, blockchain, federated learning, and Pipeline Bill of Materials (PBOM) to enhance data analytics, model development, and security across several layers, including Infrastructure, Data Lake, Federated Learning, MLOps, Data Provenance, and LLM. Each layer is designed to meet specific functional requirements, such as handling large datasets, facilitating secure federated learning, automating risk management, and ensuring data provenance and traceability. The platform, deployable in server environments (cloud or on-premises), complies with the Risk Management Framework (RMF) guidelines and security standards. It features a blockchain-enabled, coordinator-less federated learning system to enhance data privacy and security by enabling the development of privacy-preserving machine learning models with data from different wind farms. Automation plays a pivotal role throughout VindSec-Llama, with Meta’s custom-trained Llama-3 LLM used for generating remediation scripts in the Infrastructure Layer and for producing PPBOM in the MLOps Layer. The Llama-3 LLM has been quantized and fine-tuned using Qlora to ensure optimal performance on consumer-grade hardware. The MLOps pipeline setup, a critical functionality of VindSec-Llama, ensures seamless integration and deployment of machine learning models, embodying best practices in continuous integration and delivery. This setup is geared towards maximizing security, compliance, and operational efficiency. A prototype of the platform has been implemented within a wind-energy testbed with the collaboration of Department of Energy US, illustrating its practical applications and benefits.

Large Language Models (LLMs) are widely recognized for their applications in natural language processing tasks, but their potential extends far beyond traditional use cases. This paper introduces "Bassa-Llama," a novel platform that harnesses LLMs for predictive tasks in the realm of network security. Specifically, we propose a platform for real-time network attack detection in Wind Power Plants, leveraging a fine-tuned version of Meta’s Llama-3 LLM alongside blockchain and NFT-based data storage. Using a network PCAP dataset containing both malicious and benign packets, we fine-tune the Llama-3 LLM, with Quantized Low-Rank Adapter (QLoRA), to detect anomalies in network traffic. This approach ensures optimal performance on consumer-grade hardware while significantly enhancing the model’s ability to accurately analyze PCAP data and identify attack patterns. The end-to-end orchestration of the real-time network attack detection flow for Wind Power Plants is fully automated through blockchain smart contracts, and NFTs for storing identified attack data from the PCAP. To the best of our knowledge, this research represents the first effort to utilize a fine-tuned LLM for real-time network attack detection tasks. The results highlight the transformative potential of combining fine-tuned LLMs with blockchain and NFTs to build robust and secure network defense systems for Wind Power Plants. A prototype of the proposed platform was developed in collaboration with the U.S. Department of Energy, utilizing a simulated Wind Power Plant as a testbed.

The main challenges in the 5G/6G network slicing are resource sharing, network slice orchestration, and network optimization in the 5G ecosystem. This paper proposes a novel architecture for a dynamic network slice broker and marketplace named “SliceGPT” that leverages Custom-Trained OpenAI GPT-3.5 LLM, blockchain and NFTs to enable collaboration between different stakeholders in the 5G ecosystem to address these challenges. The platform enables different stakeholders in 5G network slicing (e.g., cloud providers, network operators, RAN providers, and transport network providers) to share and rent their resources to create customized network slices that meet the specific requirements of 5G applications. The orchestration of network slices is managed through blockchain smart contracts, and the resulting network slices are encoded as NFT tokens and made available for purchase in a decentralized NFT marketplace. Customers can select and purchase the network slices that best meet their needs by paying either crypto or flat currency. Revenue generated through the sale of network slices is distributed among different providers, facilitating a fair and efficient marketplace. Intelligent network slice optimization is accomplished through the utilization of a custom-trained GPT-3.5 LLM(which powers the ChatGPT). This LLM can generate valuable insights and recommendations from extensive network datasets, contributing to the optimization of network slices for enhanced performance and efficiency. A prototype of SliceGPT has been implemented with FreedomFi 5G gateway, OpenAirInterface 5G core, OpenAI GPT-3.5-turbo model, LlamaIndex and Langchain. To the best of our knowledge, this is the very first research endeavor to incorporate the GPT LLMs for optimizing 5G/6G network slicing,

Traditional and indigenous medicine, deeply rooted in ancient traditions and wisdom, plays a crucial role in global healthcare and cultural identity. These practices provide treatments for illnesses such as cancer and bone injuries, which often lack effective remedies in Western medicine. However, these valuable systems face challenges like potential knowledge loss, undervaluation of practitioners’ expertise, and the risk of fraud due to the absence of credential verification mechanisms. In this research, we introduce "WedaGPT," a Generative AI-enabled platform that utilizes a custom-trained Meta’s Llama2 Large Language Model (LLM), Blockchain, self-sovereign identity (SSI), Non-Fungible Tokens (NFTs), and model cards to share traditional medical knowledge and address these issues. WedaGPT creates a collaborative ecosystem connecting doctors, medicine providers, therapists, patients, and technology experts, all committed to preserving and advancing traditional healing practices. This platform enables secure and transparent contributions from all stakeholders to patient well-being. Ancient medical recipe books are translated into English and digitized into PDF formats to enrich the platform’s knowledge base. These texts are used to fine-tune the Llama2 LLM, which has been quantized and optimized with Qlora for performance on consumer-grade hardware. Through a chat-based interface in the SSI-enabled mobile wallet, users can interact with the LLM and access detailed information on treatments, recipes, prescriptions, and healing methods. Additionally, users can consult remotely with doctors who prescribe treatments through this wallet. A key feature of WedaGPT is transforming ancient medicinal recipes into NFT tokens for sale on NFT marketplaces, giving traditional knowledge digital authenticity and economic value. Revenue from these sales is distributed among platform contributors, promoting equitable ownership and recognition. Medical recipe data, including treatment histories and physician details, are encapsulated in Model Cards and securely stored on the blockchain. This system offers mechanisms to verify doctors and treatments in a privacy-preserving way, potentially reducing fraud and medication errors.

The network slicing in 5G/6G mobile networks enables billions of connected devices to transmit data at higher rates than ever before. The high number of devices and the huge data rates result in configuration complexities and complex security management. Machine learning techniques could play a key role in managing these system complexities. While feder-ated learning (FL) has recently been proposed as an emerging paradigm to build privacy-preserving machine learning models, many of the existing systems involve centralized coordinators which are known to be vulnerable to attacks and privacy breaches. In addition, current FL models have weak support for transparency and provenance mechanisms. In this paper, we propose a Blockchain-based, Zero-trust Security-enabled Federated Learning system “Skunk” to address privacy and data provenance requirements. The proposed federated learning system also supports the requirements of 5G/6G networks. The sharding-based architecture in the blockchain enables the deployment of Skunk in 5G/6G network slice environments. As a use case of Skunk, we have considered a scenario with IoT device attacks in a 5G/6G network. The proposed FL models detect such attacks in the 5G/6G network sliced environment.

Blockchain is a peer-to-peer distributed storage that stores chronological series of transactions in a tamper-resistant manner. Blockchain became popular in various industries due to its decentralized trust ecosystem. When integrating blockchain with big data, one encounters many challenges. Current public blockchain does not support high transaction throughput; it does not scale in terms of big data storage and management; it does not provide keyword-based search and retrieval; and so on. As a result, it is hard to incorporate existing blockchain systems for big data applications. In this research, we propose a new blockchain storage "Mystiko" that is built over the Apache Cassandra distributed database to incorporate big data. Mystiko supports high transaction throughput, high scalability, high availability and full text search features. With Mystiko, we make big data more secure, structured and meaningful, and allows further data analytics on big data to be more easily performed.

ASEAN region is currently in a unique position from the perspective of air traffic management. It comprises a large amount of independent states with limited collaboration between air traffic regulators; large aquatic zones often make air travel the only viable option, creating high demand for denser air traffic; on the other hand, there is low visibility of the air traffic over the water zones. As a result, there is a lack of the big picture of air traffic in the region, and sub-optimal air traffic configuration. This work introduces a real time tool for visual exploratory data analysis of the air traffic data that facilitates understanding of the regional air traffic dynamics. We discuss and estimate several spatial data analysis algorithms aimed at fast (real time) and crude results that are ideal for exploratory visual analysis of large spatial datasets.

Traditional cloud storage has relied almost exclusively on large storage providers, who act as trusted third parties to transfer and store data. This model poses a number of issues including data availability, high operational cost, and data security. In this paper, we introduce a system that leverages blockchain technology to provide a secure distributed data storage with keyword search service. The system allows the client to upload their data in encrypted form, distributes the data content to cloud nodes and ensures data availability using cryptographic techniques. It also provides the data owner a capability to grant permission for others to search on her data. Finally, the system supports private keyword search over the encrypted dataset.

Together with the popularity of graph structure in real world data, graph analysis has become an attractive research topic. Triangle counting is one of typical graph mining tasks and plays a significant role in complex network analysis, with a wide range of applications in social network analysis, spam detection, and computer-aided design applications. Given the sensitive nature of graph data, the disclosure of graph content for the purpose of analysis may raise a major privacy concern to both individuals and organizations. In this paper, we propose a protocol for privacy-preserving triangle counting when the overall graph is distributed to different parties. We firstly present the solution in a two-party scenario and then extend the protocol to any number of participating parties. The graph structure is represented by an adjacency matrix, and cryptographic secure matrix computation is utilized to tackle the problem. A secure multi-party power of matrix sum protocol is proposed to serve as a building block for the solution. We have evaluated the computational complexity and scalability of the proposed protocols both analytically and empirically. The results show that the protocols are efficient and scalable for small and medium size data.

There has been a significant growth in the interest to outsource data as well as operational services to clouds. Data outsourcing deprives the data owners of direct control over their data and this introduces new privacy risks. Data encryption has been introduced to address the data confidentiality issue. However, the management of encrypted data poses several challenges, the most important one is data usability in which the data owner should have the ability to query to obtain the useful data when needed. In this paper, we study the problem of supporting an important class of complex search operation: multidimensional range query. We propose a three party architecture as well as study different protocols to support multidimensional range query over encrypted remote data while remaining rigorous privacy guarantee. We examine both static and dynamic cases in which data records can be appended to the existing data set. Solutions for both fixed and unknown set of queried attributes are studied.

Recommending content to users involves understanding a) what to present and b) how to present them, so as to increase quality of experience (QoE) and thereby, content consumption. This work attempts to address the question of how to present contents in a way so that the user finds it easy to get to desired content. While the process of User Interface (UI) design is dependent on several human factors, there are basic design components and their combination that have to be common to any recommender system user interface. Personalization of the UI design process involves picking the right components and their combination, and presenting a UI to suit the usage behavior of an individual user, so as to enhance the QoE. This work proposes a system that learns from a user's content consumption patterns and makes some recommendations regarding how to present the content for the user (in the context of Video-On-Demand/Live-TV services on Computer displays), so as to enhance the QoE of the recommender system.

Big data is the biggest asset today, for organization sto amass and leverage to improve on services and profits. Knowing this, data brokers have excelled inthe task of collecting and selling user information to interested parties. This robs the users of the monetizing potential of their information. In this paper, we have put forth our idea for users to monetize on their information aided by the notion of Shannon's information theory and combined with the buyer's demand modeled on the exponential mechanism resulting in a fair and balanced pricing model for the internet information market.

This paper addresses the problem of detecting the presence of malware that leaveperiodictraces innetworktraffic. This characteristic behavior of malware was found to be surprisingly prevalent in a parallel study. To this end, we propose a visual analytics solution that supports both automatic detection and manual inspection of periodic signals hidden in network traffic. The detected periodic signals are visually verified in an overview using a circular graph and two stacked histograms as well as in detail using deep packet inspection. Our approach offers the capability to detect complex periodic patterns, but avoids the unverifiability issue often encountered in related work. The periodicity assumption imposed on malware behavior is a relatively weak assumption, but initial evaluations with a simulated scenario as well as a publicly available network capture demonstrate its applicability.

Individuals and companies, taking advantage of cloud computing which affords both resource and compute scalability, are willing to outsource their exploding data to save the storage and managing cost, however, users often do not fully trust the cloud and therefore outsource their private data after encryption to protect the data privacy. Here, as the data are both encrypted and outsourced in the cloud, how to securely and efficiently store and process such data becomes a challenging task and a primary concern. Support vector machine (SVM) classification, among different data mining and machine learning algorithms, has been very widely used in practical applications, which however, does not have a corresponding solution for such outsourced and encrypted data. Also, existing secure methods only assume that the data is locally stored by users rather than outsourced. To address this problem, we propose a novel Protocol for Outsourced SVM (POS) in this paper. POS lets cloud and users perform collaborative operations on encrypted and outsourced data without violating the data privacy contributed by each user. We formally verified that POS is correct and secure. We also conducted experimental analysis.

With the push of cloud computing which has both resource and compute scalability, data, which has been exploding in the past years, are often outsourced to a server. To this end, secure and efficient data processing and mining on outsourced private database becomes a primary concern for users. Among different secure data mining and machine learning algorithms, gradient descent method, as a widely used optimization paradigm, aims at approximating a target function to reach a local minimum, which is always deemed as a decision model to be discovered. In existing methods, users are assumed to hold and process their own data, and all users follow a secure protocol to perform gradient descent algorithm. However, such methods are not applicable to a cloud platform since that data is outsourced to a centralized server after encryption. To address this problem, we propose an Encrypted Gradient Descent Protocol (EGDP) in this paper. In EGDP, both users and server perform collaborative operations to learn and approximate the target function without violating data privacy. We formally proved that EGDP is secure and can return correct result.

Rule mining, for discovering valuable relations between items in large databases, has been a popular and well researched method for years. However, such old but important technique faces huge challenges and difficulties in the era of cloud computing although which affords both storage and computing scalability: 1) data are outsourced to a cloud due to data explosion and high storage and management cost, 2) moreover, data are usually encrypted first before being outsourced for privacy's sake. Existing privacy-preserving rule mining methods only assume a distributed model where every data owner holds the self data without encryption and together follow a secure protocol to perform rule mining. To address this limitation, we propose a novel Protocol for Outsourced Rule Mining (PORM) in this paper. PORM performs rule mining in a cloud environment where data are both encrypted and outsourced. We formally proved that PORM is both correct and secure, and we also extended PORM to the multiple-user scenario.

Thanks to the proliferation of social media and networking sites today, a massive amount of data is being generated by the second. This information is collated, collected and trawled from various sources to construct a `user profile' of sorts and used to type cast and categorize people. This is done on a regular basis by data brokers, whose sole purpose is to collect user information and sell them to interested parties. Due to the lack of any specific mechanism to ascertain the value of this information, the contributors and owners of this big data, i.e. the internet users from whom this information is collected from, do not realize the potential of their information and how valuable it is and hence go largely uncompensated. In this paper, we begin by explaining the background and problems associated with putting a price to data or information and then we put forth our idea of monetizing information by treating it like an asset and propose a transparent revenue generation model, based on the demand in the information market for various types of information. This model would offer both the consumers and buyers of information a transparent and fair mechanism to conduct business in the information market.

Today big data is synonymous with every business and organization, so much so that data brokers have made a business of trading this big data like any other commodity. In turn, the buyers of this big data make massive profits. The only one who loses out on profits and his privacy is the internet user - the generator and owner of this big data. Our work looks at allowing the user to monetize on his big data while maintaining a measure of privacy. The resulting trade off between privacy and disclosure allows the users and the buyers to decide on a suitable price for this big data.

With high and growing supply of Database-as-a-Service solutions from cloud platform vendors, many enterprises still show moderate to low demand for them. Even though migration to a DaaS solution might result in a significantly reduced bill for IT maintenance, data security and privacy issues are among the reasons of low popularity of these services. Such a migration is also often only justified if it could be done seamlessly, with as few changes to the system as possible. Transparent Data Encryption could help, but solutions for TDE shipped with major database systems are limited to securing only data-at-rest, and appear to be useless if the machine could be physically accessed by the adversary, which is a probable risk when hosting in the cloud. This paper proposes a different approach to TDE, which takes into account cloud-specific risks, extends encryption to cover data-in-use and partly data-in-motion, and is capable of executing large subsets of SQL including heavy relational operations, complex operations over attributes, and transactions.

Organizations and individuals nowadays face increasing daily operations closely rely on a huge amount of private data which is outsourced to a centralized server. Secure and efficient data processing and mining on such outsourced private data becomes a primary concern for users, especially with the push of cloud computing which has both resource and compute scalability. Among the building blocks of secure data mining algorithms, secure scalar product is used to calculate the sum of the products of the corresponding values of two vectors. Existing privacy preserving methods assume data is stored at the user side, and users follow a protocol to perform privacy preserving scalar product. However, such methods are not applicable as data now is outsourced to a centralized server in its encrypted form. To solve this problem, in this paper, we design a novel Protocol for Outsourced Scalar Product (POSP) that performs collaborative operations between server and users to produce the scalar product result without violating each user's data privacy. We proved that POSP can return the correct result and is secure. We also analysed that POSP has linear complexity in terms of space, computation, and communication with respect to the vector length.

Even though the concept of a Database-as-a-Service (DaaS) is becoming more popular and offers significant expenditure cuts, enterprises are still reluctant to migrate their data storing and processing to the cloud. One of the reasons to that is a lack of solid security guarantees. Encrypted database is one of the major approaches to address the security of cloud data processing. However, in order to provide processing capabilities over encrypted data, multiple techniques need to be combined and adjusted to work together. This paper introduces a modular and extensible framework model of an encrypted database, which makes it possible to execute a wide range of queries, including those with complex arithmetic expressions, retaining data privacy even with an adversary gaining full access to the database server. Proposed model could be used as a basis for encrypted database systems with various functional requirements.

Software released to the user has the risk of reverse engineering attacks. Software control flow obfuscation is one of the techniques used to make the reverse engineering of software programs harder. Control flow obfuscation, obscures the control flow of the program so that it is hard for an analyzer to decode the logic of the program. In this paper, we propose an obfuscation algorithm which obscures the control flow across functions. In our method code fragments from each function is stripped from the original function and is stored in another function. Each function will be having code fragments from different functions, thereby creating a function level shuffled version of the original program. Control flow is obscured between and within the function by this method. Experimental results indicate that the algorithm performs well against automated attacks.

Secure and efficient data storage and computation for an outsourced database is a primary concern for users, especially with the push for cloud computing that affords both compute and resource scalability. Among the diverse secure building blocks for secure analytical computations on outsourced databases, the encrypted set intersection operation extracts common sensitive information from datasets belonging to different users. In existing methods, each user holds their sensitive data and all users follow a secure protocol to perform set intersection. This approach is not applicable to the cloud platform, where data resides in the cloud platform in encrypted form and not at each user site. To address this limitation, in this paper, we design the Encrypted Set Intersection Protocol (ESIP) that allows server and users to perform collaborative operations to obtain the correct set intersection result without violating privacy of data contributed by each user at the server.

Software distributed can be reverse engineered using software analyzing tools, thereby letting an adversary understand the logic of the program and finding the vulnerabilities in the program. Control flow obfuscation is one of the techniques that can obscure the program so that the reverse engineering tools get an erroneous result while analyzing the program. In this paper, we propose a binary obfuscation algorithm that makes it difficult to reverse engineer the programs. The basic idea is to fragment the binary code by moving instruction sets from various parts of the program to a new code block. The control flow, to and from the new code block, is camouflaged using dynamic instruction modification during runtime. Experimental results indicate that the algorithm performs well against reverse engineering attacks by standard tools.

While DaaS is becoming more and more popular enterprises start considering it as an option to reduce IT maintenance costs. But data privacy and security issues keep most of them from moving to the cloud. Existing schemes of encrypting the database usually either provide a very basic search only (e.g., SELECT * FROM t WHERE t.a = 5) and nothing more or give users more flexibility at a cost of data security. In this paper we try to introduce a highly secure and flexible database encryption scheme allowing multiple users to perform more complex queries, including cross-table joins and still achieve acceptable performance by utilizing database's built-in indexing capabilities. Several dedicated proxy servers in between the user and the database server help to achieve this and make it all transparent for the user. Developed solution shows acceptable performance level in most of the testing cases.

The exponential growth of web services have necessitate the evolution of network infrastructures to meet this challenge. We envision the myriad of Internet connected devices coming together to provide a robust and reliable service network. We propose an Autonomous Swarm Network to provide auton

Classification in Peer-to-Peer (P2P) networks is important to many real applications, such as distributed intrusion detection, distributed recommendation systems, and distributed antispam detection. However, it is very challenging to perform classification in P2P networks due to many practical issues, such as scalability, peer dynamism, and asynchronism. This article investigates the practical techniques of constructing Support Vector Machine (SVM) classifiers in the P2P networks. In particular, we demonstrate how to efficiently cascade SVM in a P2P network with the use of reduced SVM. In addition, we propose to fuse the concept of cascade SVM with bootstrap aggregation to effectively balance the trade-off between classification accuracy, model construction, and prediction cost. We provide theoretical insights for the proposed solutions and conduct an extensive set of empirical studies on a number of large-scale datasets. Encouraging results validate the efficacy of the proposed approach.

Database-as-a-Service is becoming more and more popular for many organizations. Storing data on the cloud can significantly reduce costs in terms of maintenance costs and initial investment costs. But due to data privacy and security concerns, many companies are still reluctant to use outsourced database service. In this paper, we present a solution to protect data privacy on the cloud, at the same time, allowing multiple users accessing the shared database with security assurance.

Social networks are fundamental to virtual communities (e.g., forums, blogs) and virtual communities benefit from well-established social networks. As making friends with other members is a common way to establish social relationships and people need to decide whom they should trust when making friends, friend recommendation has received considerable attention in virtual communities. Towards this goal, we first formulate hypotheses on factors that influence trust and the probability of establishing friendships from various interaction attributes in virtual communities. Through experiments on real interaction and friendship data, we validate the proposed hypotheses and propose a novel interaction ranking-based trustworthy friend recommendation model called IntRank for recommending trustworthy friends to community members. Different from traditional friend recommendation mechanisms, IntRank is built on the foundation of carefully verified interaction attributes that influence trust and friendship probability in virtual communities. It is able to effectively recommend trustworthy friends as confirmed by the performance evaluation results.

As the amount of user generated content grows, personal information management has become a challenging problem. Several information management approaches, such as desktop search, document organization and (collaborative) document tagging have been proposed to address this, however they are either inappropriate or inefficient. Automated collaborative document tagging approaches mitigate the problems of manual tagging, but they are usually based on centralized settings which are plagued by problems such as scalability, privacy, etc. To resolve these issues, we present P2PDocTagger, an automated and distributed document tagging system based on classification in P2P networks. P2P-DocTagger minimizes the efforts of individual peers and reduces computation and communication cost while providing high tagging accuracy, and eases of document organization/retrieval. In addition, we provide a realistic and flexible simulation toolkit -- P2PDMT, to facilitate the development and testing of P2P data mining algorithms.

Recent interests on XML, the Semantic Web, and Web ontology, among other topics, have sparked a renewed interest on graph-structured databases. A fundamental query on graphs is the reachability test of nodes. Recently, 2-hop labeling has been proposed to index a large collection of XML and/or graphs for efficient reachability tests. However, there has been few work on updates of 2-hop labeling. This is compounded by the fact that data may often change over time. In response to these, this paper studies incremental maintenance of 2-hop labeling. We identify the main reason for the inefficiency of updates of existing 2-hop labels. We propose three updatable 2-hop labelings, hybrids of 2-hop labeling, and their incremental maintenance algorithms. The proposed 2-hop labeling is derived from graph connectivity, as opposed to set cover which is used by most previous works. Our experimental evaluation illustrates the space efficiency and update performance of various kinds of 2-hop labelings. Our results show that our incremental maintenance algorithm can be two orders of magnitude faster than previous methods and the size of our 2-hop labeling can be comparable to existing 2-hop labeling. We conclude that there is a natural way to spare some index size for update performance in 2-hop labeling.

Gradient descent is a widely used paradigm for solving many optimization problems. Gradient descent aims to minimize a target function in order to reach a local minimum. In machine learning or data mining, this function corresponds to a decision model that is to be discovered. In this paper, we propose a preliminary formulation of gradient descent with data privacy preservation. We present two approaches-stochastic approach and least square approach-under different assumptions. Four protocols are proposed for the two approaches incorporating various secure building blocks for both horizontally and vertically partitioned data. We conduct experiments to evaluate the scalability of the proposed secure building blocks and the accuracy and efficiency of the protocols for four different scenarios. The excremental results show that the proposed secure building blocks are reasonably scalable and the proposed protocols allow us to determine a better secure protocol for the applications for each scenario.

In this paper, we propose secure protocols to perform singular value decomposition (SVD) for two parties over horizontally and vertically partitioned data. We propose various secure building blocks for the computations of QR algorithm so that it is privacy-preserving. Some of the proposed secure building blocks include secure matrix multiplication, (x+y)-1, and radic(x+y). Together, they allow us to derive privacy-preserving SVD (PPSVD) based on a privacy-preserving QR algorithm. Finally we conduct experiments to evaluate the proposed secure building blocks and protocols. The results show that the proposed protocols for SVD achieve high accuracy for matrices of small and medium size.

In data stream clustering, it is desirable to have algorithms that are able to detect clusters of arbitrary shape, clusters that evolve over time, and clusters with noise. Existing stream data clustering algorithms are generally based on an online-offline approach: The online component captures synopsis information from the data stream (thus, overcoming real-time and memory constraints) and the offline component generates clusters using the stored synopsis. The online-offline approach affects the overall performance of stream data clustering in various ways: the ease of deriving synopsis from streaming data; the complexity of data structure for storing and managing synopsis; and the frequency at which the offline component is used to generate clusters. In this article, we propose an algorithm that (1) computes and updates synopsis information in constant time; (2) allows users to discover clusters at multiple resolutions; (3) determines the right time for users to generate clusters from the synopsis information; (4) generates clusters of higher purity than existing algorithms; and (5) determines the right threshold function for density-based clustering based on the fading model of stream data. To the best of our knowledge, no existing data stream algorithms has all of these features. Experimental results show that our algorithm is able to detect arbitrarily shaped, evolving clusters with high quality.

Currently, many privacy-preserving data mining (PPDM) algorithms assume the semi-honest model and/or malicious model of multi-party interaction. However, both models are far from able to fully capture the complexity of events and data interactions among parties in the process of secure data mining. In the paper, we study the problem of security violations when a malicious party provides false data. We identify four privacy vulnerabilities of secure scalar product protocols that underlie many current PPDM algorithms. We propose a general more model of two-party interaction and demonstrate its applicability to securely compute (x1 + y1)(x2 + y2) and (x + y)log2(x + y) where xi and yi are private values held by each party respectively. We show how the proposed model can be used to securely compute four commonly used kernel functions and other common functions. We also propose two necessary conditions and two basic measures that should be adopted in the current malicious model.

Accuracy and efficiency of clustering algorithms depend greatly on the input data. Thus, removing unimportant features from the dataset can help us form better clusters in lesser time. These unimportant features may be those that are redundant, or affected by noise, etc. We also need to consider the fact that the features we finally choose, should represent the original dataset in the best possible way. In other words, the underlying structure of the original dataset should be the same as that of the dataset that contains only the selected features. In this paper we propose a technique that selects a subset of features that best represent the entire dataset. This technique is based on two measures - Distance Measure and Similarity Measure. We first group the similar features and then select a proportionate number of features from each group. We perform experiments on a gene expression microarray dataset, and our experimental results show that using our technique as a pre-processing step significantly increases the quality of clusters generated by the underlying K-means algorithm.We also demonstrate that our approach is better than other contemporary pre-processing filters.

Recent interests on XML, Semantic Web, and Web ontology, among other topics, have sparked a renewed interest on graph-structured databases. A fundamental query on graphs is the reachability test of nodes. Recently, 2-hop labeling has been proposed to index large collections of XML and/or graphs for efficient reachability tests. However, there has been few work on updates of 2-hop labeling. This is compounded by the fact that Web data changes over time. In response to these, this paper studies the incremental maintenance of 2-hop labeling. We identify the main reason for the inefficiency of updates of existing 2-hop labels. We propose two updatable 2-hop labelings, hybrids of 2-hop labeling, and their incremental maintenance algorithms. The proposed 2-hop labeling is derived from graph connectivities, as opposed to set cover which is used by all previous work. Our experimental evaluation illustrates the space efficiency and update performance of various kinds of 2-hop labeling. The main conclusion is that there is a natural way to spare some index size for update performance in 2-hop labeling.

In order to satisfy individual customer needs with near mass production efficiency, product manufacturers need to effectively collaborate with their dynamic value chain partners along the product lifecycle process. A primary need is to develop semantic interpretations related to business processes and documents to all value chain partners. We propose a semantic Web service oriented framework to fulfil the task using OWLS. We present a product family ontology and sketches of two core services: Product configuration and product lifecycle cost estimation. We believe that with the benefits of rich semantic descriptions from OWLS, effective value chain integration can be developed-this is the key to successful mass customization.

Gradient descent is a widely used paradigm for solving many optimization problems. Stochastic gradient descent performs a series of iterations to minimize a target function in order to reach a local minimum. In machine learning or data mining, this function corresponds to a decision model that is to be discovered. The gradient descent paradigm underlies many commonly used techniques in data mining and machine learning, such as neural networks, Bayesian networks, genetic algorithms, and simulated annealing. To the best of our knowledge, there has not been any work that extends the notion of privacy preservation or secure multi-party computation to gradient-descent-based techniques. In this paper, we propose a preliminary approach to enable privacy preservation in gradient descent methods in general and demonstrate its feasibility in specific gradient descent methods.

Recent advances in high throughput experiments and annotations via published literature have provided a wealth of interaction maps of several biomolecular networks, including metabolic, protein-protein, and protein-DNA interaction networks. The architecture of these molecular networks reveals important principles of cellular organization and molecular functions. Analyzing such networks, i.e., discovering dense regions in the network, is an important way to identify protein complexes and functional modules. This task has been formulated as the problem of finding heavy subgraphs, the Heaviest k-Subgraph Problem (k-HSP), which itself is NP-hard. However, any method based on the k-HSP requires the parameter k and an exact solution of k-HSP may still end up as a "spurious" heavy subgraph, thus reducing its practicability in analyzing large scale biological networks. We proposed a new formulation, called the rank-HSP, and two dynamical systems to approximate its results. In addition, a novel metric, called the Standard deviation and Mean Ratio (SMR), is proposed for use in "spurious" heavy subgraphs to automate the discovery by setting a fixed threshold. Empirical results on both the simulated graphs and biological networks have demonstrated the efficiency and effectiveness of our proposal.

For many clustering algorithms, such as K-Means, EM, and CLOPE, there is usually a requirement to set some parameters. Often, these parameters directly or indirectly control the number of clusters, that is, k, to return. In the presence of different data characteristics and analysis contexts, it is often difficult for the user to estimate the number of clusters in the data set. This is especially true in text collections such as Web documents, images, or biological data. In an effort to improve the effectiveness of clustering, we seek the answer to a fundamental question: How can we effectively estimate the number of clusters in a given data set? We propose an efficient method based on spectra analysis of eigenvalues (not eigenvectors) of the data set as the solution to the above. We first present the relationship between a data set and its underlying spectra with theoretical and experimental results. We then show how our method is capable of suggesting a range of k that is well suited to different analysis contexts. Finally, we conclude with further empirical results to show how the answer to this fundamental question enhances the clustering process for large text collections.

In order to organise and manage geospatial and georeferenced information on the Web making them convenient for searching and browsing, a digital portal known as G-Portal has been designed and implemented. Compared to other digital libraries, G-Portal is unique for several of its features. It maintains metadata resources in XML with flexible resource schemas. Logical groupings of metadata resources as projects and layers are possible to allow the entire metadata collection to be partitioned differently for users with different information needs. These metadata resources can be displayed in both the classification-based and map-based interfaces provided by G-Portal. G-Portal further incorporates both a query module and an annotation module for users to search metadata and to create additional knowledge for sharing respectively. G-Portal also includes a resource classification module that categorizes resources into one or more hierarchical category trees based on user-defined classification schemas. This paper gives an overview of the G-Portal design and implementation. The portal features will be illustrated using a collection of high school geography examination-related resources.

Clustering of text collections is generally difficult due to its high dimensionality, heterogeneity, and large size. These characteristics compound the problem of determining the appropriate similarity space for clustering algorithms. Here, we propose to use the spectral analysis of the similarity space of a text collection to predict clustering behavior before actual clustering is performed. Spectral analysis is a technique that has been adopted across different domains to analyze the key encoding information of a system. Using spectral analysis for prediction is useful in first determining the quality of the similarity space and discovering any possible problems the selected feature set may present.

G-Portal is a Java-based digital library system for managing the metadata of geography related resources on the Web. In addition to providing a flexible repository subsystem to accommodate metadata of different formats using XML and XML Schemas, G-Portal organizes metadata into projects and layers, and supports an integrated and synchronized classification and map-based interfaces over the stored metadata. G-Portal also includes a classification subsystem that creates category structures and classifies metadata resources into categories based on user-specified classification schemas. Furthermore, G-Portal users can annotate resources and make their annotations available to others. In this paper, we describe the design and implementation of G-Portal and elaborate how Java is used to implement its features. G-Portal has been designed to be modular and some of the modules can be used as stand-alone tools. In this paper, we use UML notation to describe the detailed design of G-Portal and highlight some of the design decisions.

One common approach in hierarchical text classification involves associating classifiers with nodes in the category tree and classifying text documents in a top-down manner. Classification methods using this top-down approach can scale well and cope with changes to the category trees. However, all these methods suffer from blocking which refers to documents wrongly rejected by the classifiers at higher-levels and cannot be passed to the classifiers at lower-levels. We propose a classifier-centric performance measure known as blocking factor to determine the extent of the blocking. Three methods are proposed to address the blocking problem, namely, threshold reduction, restricted voting, and extended multiplicative. Our experiments using support vector machine (SVM) classifiers on the Reuters collection have shown that they all could reduce blocking and improve the classification accuracy. Our experiments have also shown that the Restricted Voting method delivered the best performance.

The importance of data mining is apparent with the advent of powerful data collection and storage tools; raw data is so abundant that manual analysis is no longer possible. Unfortunately, data mining problems are difficult to solve and this prompted the introduction of several novel data structures to improve mining efficiency. Here, we critically examine existing preprocessing data structures used in association rule mining for enhancing performance in an attempt to understand their strengths and weaknesses. Our analyses culminate in a practical structure called the SOTrielT (support-ordered trie itemset) and two synergistic algorithms to accompany it for the fast discovery of frequent itemsets. Experiments involving a wide range of synthetic data sets reveal that its algorithms outperform FP-growth, a recent association rule mining algorithm with excellent performance, by up to two orders of magnitude and, thus, verifying its' efficiency and viability.

With rapid increase of semistructured data over the Web, XML becomes popular to represent such data and emerges as a de facto standard for information exchange among e-business applications. Since XML data are widely distributed, new native XML database systems, which are specifically tailored for managing such data, become necessary. Unlike data in conventional databases, XML has special characteristics and constraints, thus, well-defined architectural framework for conventional databases are not completely relevant to be adopted in native databases. Current state-of-the-art native databases employed their own architectures in their database designs, as a result, they are inconsistent and there is still lack of a common architectural framework that can be referenced by future XML database designers. We aim to fulfill this requirement. In this paper, we propose a general and an extended architectural frame-works that are derived from our practical experience on designing an efficient storage for XML data while preserving desirable features and possible constraints. The validation demonstrates that the proposed framework covers the architectures employed in current state-of-the-art native database systems.

With the new global economy, manufacturing companies are focusing their efforts on the product development process which is fast emerging as a new competitive weapon. Several product development solutions allow engineers, suppliers, business partners and even customers to collaborate throughout the entire product lifecycle via the Internet. To gain an additional edge over competitors, it is vital that companies utilize Web logs to discover hidden knowledge about trends and patterns in such a cyberworld. However, existing Web log mining techniques are not designed for web logs generated by product data management processes. In this paper, we propose a method termed Product Development Miner (PDMiner) to mine such Web logs efficiently and effectively using a trie structure and sequential mining techniques. Experiments involving real Web logs show that PDMiner is both fast and practical.

We believe that, to manage Web data effectively, there is a need to build a data warehouse of Web data, i.e. a Web warehouse. In this paper, we focus on how to represent and store relevant hyperlinked Web documents effectively in a Web warehouse called Whoweda (WareHouse Of WEb DAta) for further querying and manipulation. We present a simple and general model for representing metadata, structure and content of Web documents and hyperlinks in Whoweda. We discuss node and link objects which are used to represent Web documents and hyperlinks respectively in Whoweda. These objects are first class objects in our data model called WHOM (WareHouse Object Model) which is designed to represent and manipulate Web data in the warehouse. An important feature of our model is that it represents metadata, content and structure as trees called node and link metadata trees, and node and link data trees.

Abstract Hierarchical text classification or simply hierarchical classification refers to assigning a document to one or more suitable categories from a hierarchical category space. In our literature survey, we have found that the existing hierarchical classification experiments used a variety of measures to evaluate performance. These performance measures often assume independence between categories and do not consider documents misclassified into categories that are similar or not far from the correct categories in the category tree. In this paper, we therefore propose new performance measures for hierarchical classification. The proposed performance measures consist of category similarity measures and distance‐based measures that consider the contributions of misclassified documents. Our experiments on hierarchical classification methods based on SVM classifiers and binary Naïve Bayes classifiers showed that SVM classifiers perform better than Naïve Bayes classifiers on Reuters‐21578 collection according to the extended measures. A new classifier‐centric measure called blocking measure is also defined to examine the performance of subtree classifiers in a top‐down level‐based hierarchical classification method.

In this paper, we present a mechanism for detecting and representing changes, given the old and new versions of a set of interlinked Web documents, retrieved in response to a user's query. In particular, we show how to detect and represent Web deltas, i.e., changes in the Web documents that are relevant to a user's query in the context of our Web warehousing system called WHOWEDA (Warehouse of Web Data). In WHOWEDA, Web information is materialized views stored in Web tables in the form of Web tuples. These Web tuples, represented as directed graphs, can be manipulated using a set of Web algebraic operators. In this paper, we present a mechanism to detect relevant Web deltas using Web algebraic operators such as the Web join and the outer Web join. Web join is used to detect identical documents residing in two Web tables, whereas, outer Web join, a derivative of Web join, is used to identify dangling Web tuples. We show how to represent these changes using delta Web tables. We develop formal algorithms for the generation of delta Web tables identifying Web documents which have been added, deleted, or modified since the last query.

We propose a k-NN query processing algorithm that uses one or more window queries to retrieve the nearest neighbors of a given query point. We also propose two different methods to estimate the ranges to be used by the window queries. Each range estimation method requires different statistical knowledge about the spatial databases. Our experiments on the TIGER data allow us to study the behavior of the proposed algorithm using different range estimation methods. Apart from not requiring direct access to the spatial indices, the window queries used in the proposed algorithm can be easily supported by non-spatial database systems containing spatial objects.

The rising popularity of electronic commerce makes data mining an indispensable technology for business competitiveness. The World Wide Web provides abundant raw data in the form of Web access logs, Web transaction logs and Web user profiles. Without data mining tools, it is impossible to make any sense of such massive data. We focus on Web usage mining because it deals most appropriately with understanding user behavioral patterns which is the key to successful customer relationship management. Previous work dealt separately with specific issues of Web usage mining and made assumptions without taking a holistic view and thus, had limited practical applicability. We formulate a novel and more holistic version of Web usage mining termed transactionized logfile mining (TRALOM) to effectively and correctly identify transactions as well as to mine useful knowledge from Web access logs. We also introduce a new data structure, called the WebTrie, to efficiently hold useful preprocessed data so that TRALOM can be done in an online and incremental fashion. Experiments conducted on real Web server logs verify the usefulness and practicality of our proposed techniques.

In any electronic commerce system, the heterogeneity of product descriptions is a critical impediment to efficient business information exchange. In the ABECOS electronic commerce system, buyer agents, seller agents, and directory agents liaise with one another in e-commerce activities. Only when agents have a common ontology of product descriptions (also called product schemas) are they able to interact seamlessly in e-commerce activities. This gives rise to the product schema integration problem (PSI); the problem of integrating heterogeneous schemas of a certain product into one globally compatible schema. We adopt an integration approach based on product attribute synonyms. We give a formal definition of the problem and show that it is NP-complete. We contrast our approach of study to conventional schema integration in federated databases. We also propose a set of approximate algorithms for PSI and evaluate their performance.

Association rule mining is a well-researched area where many algorithms have been proposed to improve the speed of mining. In this paper, we propose an innovative algorithm called Rapid Association Rule Mining (RARM) to once again break this speed barrier. It uses a versatile tree structure known as the Support-Ordered Trie Itemset (SOTrieIT) structure to hold pre-processed transactional data. This allows RARM to generate large 1-itemsets and 2-itemsets quickly without scanning the database and without candidate 2-itemset generation. It achieves significant speed-ups because the main bottleneck in association rule mining using the Apriori property is the generation of candidate 2-itemsets. RARM has been compared with the classical mining algorithm Apriori and it is found that it outperforms Apriori by up to two orders of magnitude (100 times), much more than what recent mining algorithms are able to achieve.

The popularity of WWW has made it a prime vehicle for disseminating information. Compared to pure static and dynamic Web pages which are dynamically created by a CGI script at run-time, we focus on semi-dynamic Web pages whose contents are extracted from source databases and they change in response to updates to the source databases. An example of such a page can be found at http://www.fish.com.sg where a list of stock information is refreshed frequently with respect to updates to base data. A crucial problem arises when base data change frequently and there is a need to keep a large set of semi-dynamic pages up-to-date in response to source changes since no one is interested in stale data on the Web; an investor may suffer great losses relying on obsolete stock price on the Web. To differentiate between the contents derived from base data and the trivial contents of a Web page, we refer to that portion of a page derived from base data as a Web view.

At present, there are no association rule mining algorithms that are suitable for use in electronic commerce because they do not consider that new products are introduced and old ones are retired frequently and they assume that support thresholds do not change. In this paper a new algorithm called Fast Online Dynamic Association Rule Mining (FOLDARM) is introduced for mining in electronic commerce. It uses a novel tree structure known as a Support-Ordered Trie Itemset (SOTrieIT) structure to hold pre-processed transactional data. It allows FOLDARM to generate large 1-itemsets and 2-itemsets quickly without scanning the database. In addition, the SOTrieIT structure can be easily and quickly updated when transactions are added or removed. It also stores data that is independent of the support threshold and thus can be used for mining with varying support thresholds without any degradation in performance. Experiments have shown that FOLDARM outperforms Apriori, a classic mining algorithm, by up to two orders of magnitude (100 times).

We show how to detect and represent Web deltas, i.e., changes in Web information, that are relevant to a user's query in the context of our Web warehousing system called WHOWEDA (Warehouse of Web Data). In WHOWEDA, Web information are materialized views stored in Web tables and can be manipulated and analyzed using a set of Web algebraic operators. We present a mechanism to detect relevant Web deltas using Web join and outer Web join. We show how to represent these changes using delta Web tables.

This paper provides a quantitative analysis of reducing cognitive overheads in a Web warehouse using an important class of operation called reverse osmosis. The analysis is used to examine two different cognitive overheads of locating relevant nodes or information and display time of a Web table. A reverse-osmosis operation enables us to eliminate in relevant information from a collection of Web documents stored in the form of a Web table. We call such an operation reverse-osmosis because it is analogous to the reverse osmosis process in the field of water purification. We discuss a formal algorithm of the reverse-osmosis operation.

With the rapid increase in the number of E-commerce Web sites, Internet users are now able to purchase items from the Web. Nevertheless, the large number of E-commerce Web sites also brings about the problem of identifying sites that sell the desired products wanted by a potential buyer. To avoid broadcasting the same queries on products to large set of databases maintained at different E-commerce Web sites, buyers can rely on a query routing system that matches the queries with the E-commerce database content. Much of the previous work in query routing has focused on databases that are document collections and queries that involve text attributes only. We propose a new database selection technique designed for distributed E-commerce databases. The technique deals with queries that involve attributes of different data types. We have incorporated the proposed technique into an agent-based query routing system known as eBroker. The performance of e-Broker for the E-commerce databases of 5 online CD stores is reported.

Although much work has been done on WWW information management, research in keeping information on Web sites up-to-date is still in its infancy. For a very large Web site whose content is derived from some source databases, it is important to maintain its freshness in response to changes to the base data. This issue is particularly significant for Web sites presenting fast-changing information, such as stock exchange information and product information. In this paper, we propose a model for studying the problem of refreshing a very large Web site in time. We identify several important research issues in maintaining the currency of information on such Web sites. We also discuss possible approaches to the timely refreshing of a Web site under the framework.

The effective representation and manipulation of web data is currently an active area of research in databases. In a data warehouse specially designed for web information, web information coupling provides the means to derive useful information from the WWW. Web information is materialized in the form of web tuples and stored in web tables. In this paper, we discuss web data visualization operators such as web nest, web unnest, web coalesce, web expand, web pack, web unpack and web sort in the context of our web warehousing system called Whoweda (Warehouse of Web Data) to provide users with the flexibility to view sets of web documents in perspectives which may be more meaningful.

A query routing broker is a software agent that determines from a large set of accessing information sources the ones most relevant to a user's information need. As the number of information sources on the Internet increases dramatically, future users will have to rely on query routing brokers to decide a small number of information sources to query without incurring too much query processing overheads. In this paper, we describe a query routing broker known as ZBroker developed for bibliographic database servers that support the Z39.50 protocol. ZBroker samples the content of each bibliographic database by using training queries and their results, and summarizes the bibliographic database content into a knowledge base. We present the design and implementation of ZBroker and describe its Web-based user interface.

With the enormous amount of data stored in the World Wide Web, it is increasingly important to design and develop powerful web warehousing tools. The key objective of our web warehousing project, called WHOWEDA (Warehouse of Web Data), is to design and implement a web warehouse that materializes and manages useful information from the web. We introduce the concept of /spl Pi/-web join in the context of WHOWEDA. /spl Pi/-web join operator is a web information manipulation operator to combine relevant web information residing in two web tables. Informally, it is the combination of web join and web project operators which filter out irrelevant information from a joined web table. We show how to construct the /spl Pi/-joined web table and its schema. We also highlight the benefits of the /spl Pi/-web join operator.

Sets and bags are closely related structures and have been studied in relational databases. A bag is different from a set in that it is sensitive to the number of times an element occurs, while a set is not. In this paper, we introduce the concept of a Web bag in the context of a World Wide Web warehouse called WHOWEDA (WareHouse Of WEb DAta) which we are currently building. Informally, a Web bag is a Web table which allows multiple occurrences of identical Web types. A Web bag helps one to discover useful knowledge from a Web table, such as visible documents or Web sites (i.e. documents/sites which can be reached by many paths), luminous documents (i.e. documents with many outgoing links) and luminous paths (i.e. frequently traversed paths). In this paper, we provide a cost-benefit analysis of materializing Web bags as compared to Web tables with distinct Web tuples.

Disk I/O has long been a performance bottleneck for very large databases. Database compression can be used to reduce disk I/O bandwidth requirements for large data transfers. The authors explore the compression of large statistical databases and propose techniques for organizing the compressed data such that standard database operations such as retrievals, inserts, deletes and modifications are supported. They examine the applicability and performance of three methods. Two of these are adaptions of existing methods, but the third, called tuple differential coding (TDC), is a new method that allows conventional access mechanisms to be used with the compressed data to provide efficient access. They demonstrate how the performance of queries that involve large data transfers can be improved with these database compression techniques.

The design of vector quantizers (VQs) that yield minimal distortion is one of the most challenging problems in source coding. The problem of VQ design is to find a codebook that gives the least overall distortion (or equivalently, the largest signal-to-noise ratio (SNR)) for a given set of input vectors. This problem is known to be difficult as there are no known closed-form solutions. The generalized Lloyd algorithm (GLA) uses a finite set of training sequences as the data source and employs an iterative refinement. Given an initial codebook, the algorithm computes the nearest focally optimum codebook only. Genetic algorithms (GAs) are emerging as widely accepted optimization and search methods. These search methods are rooted in the mechanisms of evolution and natural genetics. They have a high probability of locating the globally optimal solution in a multimodal search space. A genetic algorithmic (GA) approach to vector quantizer design that combines the GLA is presented. We refer to this hybrid as the genetic generalized Lloyd algorithm (GGLA). It works briefly as follows. Initially, a finite number of codebooks, called chromosomes, are selected. In contrast to the GLA which refines only one codebook at a time, those codebooks undergo iterative cycles of reproduction in parallel. During an iteration, each codebook is updated by GLA or GA operations (i.e., mutation, crossover, and chromosome replacement). Three versions of the GGLAs are investigated depending on how the GLA or GA is selected.

Given a statistical database containing a set of summary tables, this paper examines the complexity of retrieving data from the database in order to satisfy a query.In particular, we consider the case when the query cannot be directly satisfied via a single summary table and requires two or more summary tables.We show that a system of linear equations can be constructed from a set of summary tables whose solution(s) satisfy a query in oarying degrees.We

One approach to distributed mutual exclusion algorithms is the use of quorums. Quorum-based algorithms offer the advantage of protocol symmetry, spreading effort and responsibility uniformly across the distributed system. In this paper, we present an O(logn) algorithm to generate coterie templates of near-optimal O(n/sup 0.63/) size. Coterie templates are generic quorum structures that exhibit several desirable properties such as fault tolerance, symmetry and low storage cost. In addition, coteries can be instantiated from the template to reflect desirable network characteristics.

Data compression is one way to alleviate the I/O bottleneck problem faced by I/O-intensive applications such as databases. However, this approach is not widely used because of the lack of suitable database compression techniques. In this paper, we design and implement a novel database compression technique based on vector quantization (VQ). VQ is a data compression technique with wide applicability in speech and image coding, but it is not directly suitable for databases because it is lossy. We show how one may use a lossless version of vector quantization to reduce database space storage requirements and improve disk I/O bandwidth.>

A common approach to improving the performance of statistical query processing is to use precomputed results. Another lower-level approach would be to redesign the storage structure for statistical databases. This avenue is relatively unexplored. The objective of this paper is to present a physical storage structure for statistical databases, whose design is motivated by the characteristics of statistical queries. We show that our proposal enhances multi-attribute clustering efficiency, and improves the performance of statistical and aggregational queries. This customized structure reduces the amount of I/O incurred statistical query processing, thus decreasing the response time.>