While load balancing in distributed systems is extensively studied, traditional algorithms often assume static task loads, a simplification that fails under realistic dynamic conditions where rapid load fluctuations undermine balancing effectiveness. Addressing this critical gap, this paper introduces a novel approach by explicitly incorporating load uncertainty. We uniquely reformulate the load balancing problem using statistical performance aggregates—specifically, both mean and variance—to simultaneously optimize for balance and robustness. We propose a fully decentralized algorithm enabling nodes to collaboratively achieve this dual objective using these aggregates. Crucially, our algorithm offers mathematically provable guarantees, ensuring convergence to a global ideal state in logarithmic rounds even amidst varying loads, a guarantee often missing for dynamic scenarios. Simulations using real-world traces validate our approach, demonstrating significantly superior performance robustness and tighter load balancing compared to existing methods that rely solely on instantaneous or static metrics.

Apache Kafka is a distributed data streaming platform that is widely adopted in the industry. Producers introduce messages to Kafka server nodes, and consumers fetch interested messages in real time. In Kafka, the loads of nodes that serve streaming traffic may be imbalanced because some traffic is relatively popular, resulting in load imbalance. Balancing loads among nodes enables producers and consumers to avoid performance bottleneck, shortening the latencies of sending and receiving their messages and thus improving quality of service. Partitions are the fundamental entities that serve loads in Kafka and are responsible for hosting message payloads. Kafka provides a built-in load balancer that addresses load imbalance. The built-in load balancer distributes partitions evenly to the system, which is not designed to deal with load imbalance due to the heterogeneity of loads. The state-of-the-art load balancer released by LinkedIn, namely, Cruise Control (CC), complements the Kafka built-in load balancer. The performance quality of CC is sensitive and highly depends on the ordering of performance metric constraints. We propose Yet Another (YA) load balancer for Kafka, aiming to achieve simplicity and robustness for the average case. We compare YA with the Kafka built-in load balancer and CC in a real cluster environment. Performance results validate our findings and indicate that load balancers based on gathered performance metrics can effectively reduce the end-to-end delay perceived by streaming applications up to a ratio of approximately 20. CC and our proposed load balancer clearly outperform the Kafka built-in load balancer, yet while our proposed load balancer is comparable to CC, it is more robust.

In the Laser Powder Bed Fusion (L-PBF) process, 3D components with complex geometries are fabricated in a layer-by-layer fashion by using a controlled laser beam to selectively melt particular regions of the metal powder bed. However, due to the stochastic nature of the L-PBF process, the top surface roughness of each solidified layer tends to be different even when the optimal processing conditions for the different positions on the build plate are employed. As a result, the mechanical properties of the built components frequently vary from one component to the next. Accordingly, this study proposes an Intelligent Additive Manufacturing Architecture (IAMA) for controlling the surface roughness of each build layer through an appropriate adjustment of the laser re-melting parameters. The IAMA architecture comprises five modules, namely In-Situ Metrology (ISM), Ex-Situ Metrology (ESM), Automatic Virtual Metrology (AVM), Additive Manufacturing Simulation (AMS) and Intelligent Compensator (IC). The feasibility of the proposed architecture is demonstrated by comparing the top surface roughness of cubic and mechanical strengths of tensile test samples built using the proposed method with those built using a traditional L-PBF approach without surface roughness control. It is found that the samples fabricated using the IAMA approach have an average top surface roughness of$1.6~\mu \text{m}$and a standard deviation is$0.7~\mu \text{m}$. By contrast, the samples produced using the traditional L-PBF approach have an average surface roughness of$13.45~\mu \text{m}$and a standard deviation of$2.5~\mu \text{m}$. In addition, the specimens produced with the assistance of IAMA architecture have an average tensile strength of 1013 MPa with a standard deviation of 69.5 MPa, while those printed without surface roughness control have an average tensile strength of 903 MPa with a standard deviation of 101.4 MPa Note to Practitioners—As L-PBF produce part in a layer-by-layer manner, therefore, the roughness on the top surface of previous layer have a strong influence on the printing quality of current layer. The variations of surface roughness will lead to the fluctuation of mechanical properties of the fabricated components. Additionally, to the best of author’s knowledge, current commercial L-PBF machines can not actively control the surface roughness of samples during the process. The proposed IAMA in this work can predict and control the roughness on the top surface of each layer during L-PBF process. Therefore, the constructed architecture has strong potential for integrating into the commercial L-PBF machine for controlling the surface roughness on the top of the parts during the manufacturing process. As a result, the quality of the fabricated components is expected to be in consistent. Accordingly, L-PBF machine equipped with IAMA will have a strong potential in applying for mass production of the components in aerospace and automobile industries.

Not Only SQL (NoSQL) is a critical technology that is scalable and provides flexible schemas, thereby complementing existing relational database technologies. Although NoSQL is flourishing, present solutions lack the features required by enterprises for critical missions. In this paper, we explore solutions to the data recovery issue in NoSQL. Data recovery for any database table entails restoring the table to a prior state or replaying (insert/update) operations over the table given a time period in the past. Recovery of NoSQL database tables enables applications such as failure recovery, analysis for historical data, debugging, and auditing. In this paper, we first identify the design and implementation issues with regard to the data recovery problem for NoSQL databases, including time length of recovery, fault tolerance, scalability, memory constraint, software compatibility, and quality of recovery. Particularly, our study emphasizes on columnar NoSQL databases. We then propose and evaluate four solutions to address the data recovery problem in NoSQL; each solution has its pros and cons. We implement our solutions based on Apache HBase, a popular NoSQL database in the Hadoop ecosystem widely adopted by industry. Our implementations are extensively benchmarked with an industrial NoSQL benchmark under real environments. Specifically, our research findings and implementations in this paper have been contributed to and integrated with Apache HBase for global distribution.

Many core technologies of Industry 4.0 have gained substantial advancement in recent years. Digital Twin (DT) has become the key technology and tool for manufacturing industries to realize intelligent cyber-physical integration and digital transformation by leveraging these technologies. Although there have been many DT-related works, there is no standard definition, unified framework, and implementation approach of DT until now. Widely developing DTs for the manufacturing industry is still challenging. Thus, this paper proposes a novel implementation framework of digital twins for intelligent manufacturing, denoted as IF-DTiM, which possesses several distinct merits to distinguish itself from previous works. First, IF-DTiM fully utilizes new-generation container technology so that DT-related applications and services can be packaged in a self-contained way, rapidly deployed, and robustly operated with the capabilities of failover, autoscaling, and load balancing. Second, it leverages existing intelligent cloud manufacturing services to realize the intelligence for DT externally in a scalable and plug-and-play manner instead of using traditional approaches to embed intelligence in DT. Third, IF-DTiM contains Product DT for products, Equipment DT (i.e., EQ DT) for equipment, and Process DT for production lines, which can generically fulfill the demands and scenarios to achieve intelligent manufacturing for various manufacturing industries. Testing results show that IF-DTiM can achieve remarkable performance in rapid deployment and real-time data exchanges of DT-related applications. Finally, we develop an example DTiM system for CNC machining based on IF-DTiM to demonstrate its efficacy and applicability in facilitating the manufacturing industry to build their DT systems.Note to Practitioners—Developing Digital Twin (DT) systems to realize intelligent manufacturing is challenging. The proposed IF-DTiM (Implementation Framework of Digital Twins for Intelligent Manufacturing) provides a novel container-technology and cloud-manufacturing-service-based systematic methodology for building DTiM. In this paper, we present the system architecture and several operational scenarios (e.g., how to create and use DTs) of IF-DTiM, together with the design of its core functional mechanisms (e.g., rapid deployment scheme for DT, real-time data exchange for DT, DT interface pattern, and general workflow architecture for DT). Also, an example DTiM system for CNC machining based on IF-DTiM is presented to facilitate the practitioners to adopt the designs and niches in IF-DTiM to build their desired DTiM systems.

We present in this paper two novel infrastructural services based on Hadoop for big data storage and computing in a Taiwan's semiconductor wafer fabrication foundry. The two services include Hadoop data service (HDS) and distributed R language computing service (DRS), which have been built and operated in production systems for 3.5 years. They evolve over time by incrementally accommodating users' requirements. HDS is a web- based distributed big data storage facility. Users simply rely on HDS to access data objects stored in Hadoop with the HTTP protocol. In addition, HDS is scalable and reliable. Moreover, HDS is efficient and effective by intelligently selecting either Hadoop distributed file system (HDFS) or database (HBase) for publishing data objects. Specifically, HDS is transparent to existing analytics and data inquiry applications, such as Spark and Hive. While HDS is a unified storage for supporting sequential and random data accesses for big data in the wafer fabrication foundry, DRS is a distributed computing framework for typical R language users. R users employ DRS to enjoy data-parallel computations, effortlessly and seamlessly. Similar to HDS, DRS can be horizontally scaled out. It guarantees the completion of computational jobs even with failures. In particular, it adaptively reallocates computational resources on the fly, minimizing job execution time and maximizing utilization of allocated resources. This paper discusses the design and implementation features for HDS and DRS. It also demonstrates their performance metrics.

We present in this paper a novel infrastructural service based on Hadoop for big data storage and computing in a Taiwan's semiconductor wafer fabrication foundry. The service is named Hadoop data service (HDS), which has been built and operated in production systems for 3.5 years. It evolves over time by incrementally accommodating users' requirements. HDS is a web-based distributed big data storage facility. Users simply rely on HDS to access data objects stored in Hadoop with the HTTP protocol. In addition, HDS is scalable and reliable. Moreover, HDS is efficient and effective by intelligently selecting either Hadoop distributed file system (HDFS) or database (HBase) for publishing data objects. Specifically, HDS is transparent to existing analytics and data inquiry applications, such as Spark and Hive. This paper discusses the design and implementation features for HDS. The performance metrics of HDS are also demonstrated.

Cloud computing has become increasingly popular model for delivering applications hosted in large data centers as subscription oriented services. Hadoop is a popular system supporting the MapReduce function, which plays a crucial role in cloud computing. The resources required for executing jobs in a large data center vary according to the job type. In Hadoop, jobs are scheduled by default on a first-come-first-served basis, which may unbalance resource utilization. This paper proposes a job scheduler called the job allocation scheduler (JAS), designed to balance resource utilization. For various job workloads, the JAS categorizes jobs and then assigns tasks to a CPU-bound queue or an I/O-bound queue. However, the JAS exhibited a locality problem, which was addressed by developing a modified JAS called the job allocation scheduler with locality (JASL). The JASL improved the use of nodes and the performance of Hadoop in heterogeneous computing environments. Finally, two parameters were added to the JASL to detect inaccurate slot settings and create a dynamic job allocation scheduler with locality (DJASL). The DJASL exhibited superior performance than did the JAS, and data locality similar to that of the JASL.

With the need of data storage increases tremendously nowadays, distributed file system becomes the most important data storage system in cloud computing. In distributed file system development, there are many researchers work hard to refine the architecture to provide scalability and reliability. In our work, we propose a distributed metaserver system including metaserver scale-out, metadata replication, metaserver recovery, and metaserver management recovery mechanisms. In our experiments, the proposed system can increase the capacity of metadata and increase the reliability by fault tolerance mechanism. The overhead of read/write data is very little in the proposed system as well.

Present NoSQL databases are passive entities, where users proactively access the databases. While NoSQL databases are scalable due to their horizontal scale-out designs, data items stored in potentially very large databases are difficult to retrieve in terms of access delay, programmability and usability. In this paper, we advocate supporting events in NoSQL. By introducing publishers and subscribers to NoSQL, our proposed NoSQL data store is capable of delivering data items that users are interested in. Additionally, our NoSQL database decouples publishers and subscribers such that application developers can emphasize on data manipulation without paying attention to communications for delivering data. We formally discuss the design requirements for streaming in NoSQL, and present a prototype implementation that addresses the design issues. We also outline our ongoing works in this paper.

Peers participating in a distributed hash table (DHT) may host different numbers of virtual servers and are enabled to balance their loads in the reallocation of virtual servers. Most decentralized load balance algorithms designed for DHTs based on virtual servers require the participating peers to be asymmetric, where some serve as the rendezvous nodes to pair virtual servers and participating peers, thereby introducing another load imbalance problem. While state-of-the-art studies intend to present symmetric load balancing algorithms, they introduce significant algorithmic overheads and guarantee no rigorous performance metrics. In this paper, a novel symmetric load balancing algorithm for DHTs is presented by having the participating peers approximate the system state with histograms and cooperatively implement a global index. Each peer independently reallocates in our proposal its locally hosted virtual servers by publishing and inquiring the global index based on their histograms. Unlike competitive algorithms, our proposal exhibits analytical performance guarantees in terms of the load balance factor and the algorithmic convergence rate, and introduces no load imbalance problem due to the algorithmic workload. Through computer simulations, we show that our proposal clearly outperforms existing distributed algorithms in terms of load balance factor with a comparable movement cost.

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

Distributed file systems (DFS) are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions, a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance, that is, the file chunks are not distributed as uniformly as possible in the nodes. Although distributed load balancing algorithms exist in the literature to deal with the load imbalance problem, emerging DFSs in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, we illustrate and define the load rebalancing problem in cloud DFSs. We advocate file systems in clouds shall incorporate decentralized load rebalancing algorithms to eliminate the performance bottleneck and the single point of failure. Simulation results for a potential distributed load balancing algorithm are illustrated. The performance of our proposal implemented in the Hadoop distributed file system is also demonstrated.

Unstructured peer-to-peer (P2P) file-sharing networks are popular in the mass market. As the peers participating in unstructured networks interconnect randomly, they rely on flooding query messages to discover objects of interest and thus introduce remarkable network traffic. Empirical measurement studies indicate that the peers in P2P networks have similar preferences, and have recently proposed unstructured P2P networks that organize participating peers by exploiting their similarity. The resultant networks may not perform searches efficiently and effectively because existing overlay topology construction algorithms often create unstructured P2P networks without performance guarantees. Thus, we propose a novel overlay formation algorithm for unstructured P2P networks. Based on the file sharing pattern exhibiting the power-law property, our proposal is unique in that it poses rigorous performance guarantees. Theoretical performance results conclude that in a constant probability, 1) searching an object in our proposed network efficiently takes O(\ln^{c} {\cal N}) hops (where c is a small constant), and 2) the search progressively and effectively exploits the similarity of peers. In addition, the success ratio of discovering an object approximates 100 percent. We validate our theoretical analysis and compare our proposal to competing algorithms in simulations. Based on the simulation results, our proposal clearly outperforms the competing algorithms in terms of 1) the hop count of routing a query message, 2) the successful ratio of resolving a query, 3) the number of messages required for resolving a query, and 4) the message overhead for maintaining and formatting the overlay.

With the notion of virtual servers, peers participating in a distributed hash table (DHT) may host different numbers of virtual servers and are enabled to balance their loads in the reallocation of virtual servers. While most decentralized load balance algorithms designed for the DHTs based on virtual servers offer no performance guarantees, we present in this paper a novel distributed load balancing algorithm for DHTs with stochastic performance bounds. In addition to rigorous performance analysis, we compare our proposal with competitive algorithms through computer simulations. The simulation results indicate that our proposal clearly outperforms the previous algorithms.

With the notion of virtual servers, peers participating in a heterogeneous, structured peer-to-peer (P2P) network may host different numbers of virtual servers, and by migrating virtual servers, peers can balance their loads proportional to their capacities. The existing and decentralized load balance algorithms designed for the heterogeneous, structured P2P networks either explicitly construct auxiliary networks to manipulate global information or implicitly demand the P2P substrates organized in a hierarchical fashion. Without relying on any auxiliary networks and independent of the geometry of the P2P substrates, we present, in this paper, a novel load balancing algorithm that is unique in that each participating peer is based on the partial knowledge of the system to estimate the probability distributions of the capacities of peers and the loads of virtual servers, resulting in imperfect knowledge of the system state. With the imperfect system state, peers can compute their expected loads and reallocate their loads in parallel. Through extensive simulations, we compare our proposal to prior load balancing algorithms.

Unstructured peer-to-peer (P2P) file-sharing networks are popular in the mass market. As the peers participating in unstructured networks interconnect randomly, they rely on flooding query messages to discover objects of interest. Empirical measurement studies indicate that the peers in P2P networks have similar preferences, and recently proposed unstructured P2P networks intend to organize the participating peers in a small-world (SW) fashion by exploiting the knowledge of contents stored in peers. As existing algorithms for constructing SW-based unstructured P2P networks may not precisely reveal the object sharing patterns, the resultant networks thus may not perform searches efficiently and effectively by exploiting the common interests among peers. In this paper, we suggest a novel P2P network formation algorithm to construct SW-based unstructured networks. We validate our proposal in simulations with an empirical data set, and the simulation results prove that our proposal greatly outperforms existing algorithms in terms of search efficiency and effectiveness.

In an unstructured peer-to-peer (P2P) network (e.g., Gnutella), participating peers choose their neighbors randomly such that the resultant P2P network mismatches its underlying physical network, resulting in the lengthy communication between the peers and redundant network traffics generated in the underlying network. Previous solutions to the topology mismatch problem in the literature either have no performance guarantees or are far from the optimum. In this paper, we propose a novel topology matching algorithm based on the Metropolis-Hastings method. Our proposal is guided by our insight analytical model and is close to the optimal design. Specifically, we show that our proposal constructs an unstructured P2P network in which a broadcast message, originated by any node v, reaches any other node u by taking approximately the only physical end-to-end delay between v and u. In addition, our design guarantees the exponential broadcast scope. We verify our solution through extensive simulations and show that our proposal considerably outperforms state-of-the-art solutions.

In an unstructured peer-to-peer (P2P) network (e.g., Gnutella), participating peers choose their neighbors randomly such that the resultant P2P network mismatches its underlying physical network, resulting in the lengthy communication between the peers and redundant network traffics generated in the underlying network. Previous solutions to the topology-mismatch problem in the literature are far from the optimum. In this paper, we propose a novel topology-matching algorithm based on the Metropolis-Hastings method. Through extensive simulations, we show that our proposal constructs an unstructured P2P network where a broadcast message, originated by any node v, reaches any other node u by taking approximately the only physical end-to-end delay between v and u.

Prior studies show that more than 70 percent of communication paths in a popular unstructured peer-to-peer (P2P) system (i.e., Gnutella) do not exploit the physical network topology, leading to the topology mismatch problem, and thus, lengthen communication between participating peers. While previous efforts in solving overlay topology matching problems do not guarantee the bounds of performance metrics (e.g., the communication delay between any two overlay peers and the broadcasting scope of any participating peer), in this paper, we present a novel topology matching algorithm that has provable performance qualities. In our proposal, each participating node creates and manages a constant number of overlay connections to other peers in a distributed manner. In rigorous performance analysis, we show that 1) the expected overlay communication delay between any two nodes in our P2P network is a constant; 2) in addition, any joining node has the exponential broadcasting scope in expectation; 3) furthermore, a participating node takes a polylogarithmic overhead to exploit the physical network locality and maintain its flooding scope. Together with extensive simulations, we present our proposal that significantly outperforms two recent solutions, i.e., THANCS and mOverlay, in terms of overlay communication latency and/or broadcasting scope.

Small-world (SW) networks possess two properties, namely low diameter and high clustering coefficient, which are often desired by large-scale peer-to-peer networks. Prior studies have shown that the construction of an SW network can be based on a d-regular graph, and each node in the graph maintains d local neighbors and a small constant number of long-distance contacts. However, it is commonly understood that it is difficult to construct a short route in an SW network, given source (s) and target (t) nodes, though an SW network guarantees that a short route from s to t exists. Prior work in [1] proposed a navigable SW network for a d-dimensional lattice such that a simple localized routing algorithm can be devised to route a message from s to t using O(log2,X) hops, where X is the number of nodes in the network. In this paper, we present a novel navigable SW network based on a hierarchical model. Compared to previous efforts, the novelty of our study presents 1) that our network construction based on a hierarchical model is decentralized, 2) that routing a message between any two nodes in our SW network takes logarithmic hopcount in expectation, 3) that our SW network has high cluster coefficient, and 4) that the performance of our proposal is mathematically provable. We support the performance of our proposal in this study through rigorous, thorough performance analysis and extensive simulations.

Virtual servers provide an abstraction layer to exploit the heterogeneity of peers (or nodes) that participate in a peer-to-peer (P2P) network based on the distributed hash table (DHT). In this paper, we present a load-balanced DHT network based on virtual servers. In our proposal, participating peers accept loads proportional to their capacities. By sampling the capacities of peers and workloads of virtual servers in the system, our design approximates the probability distributions for the capacities and workloads. Each node is based on the probability distributions to match their local virtual servers and other peers. Our proposal is developed through rigorous performance analysis, and is evaluated in extensive simulations.

Peer-to-peer (P2P) networks often demand scalability, low communication latency among nodes, and low systemwide overhead. For scalability, a node maintains partial states of a P2P network and connects to a few nodes. For fast communication, a P2P network intends to reduce the communication latency between any two nodes as much as possible. With regard to a low systemwide overhead, a P2P network minimizes its traffic in maintaining its performance efficiency and functional correctness. In this paper, we present a novel tree-based P2P network with low communication delay and low systemwide overhead. The merits of our tree-based network include 1) a tree-shaped P2P network, which guarantees that the degree of a node is constant in probability, regardless of the system size (the network diameter in our tree-based network increases logarithmically with an increase in the system size, and in particular, given a physical network with a power-law latency expansion property, we show that the diameter of our tree network is constant), and 2) provable performance guarantees. We evaluate our proposal by a rigorous performance analysis, and we validate this by extensive simulations.

Scoped broadcast disseminates a message to all the nodes within a designated physical/logical region in an overlay network. It can be a basic building block for applications such as information search, data broadcasting, and overlay structure diagnostic. In this paper, we study scoped broadcast in peer-to-peer (P2P) overlay networks based on distributed hash tables (DHT). Since P2P networks have very dynamic behavior due to peer joining and departure, it is interesting to know how many peers can be reached with one scoped broadcast. This depends mainly on the cost we would like to pay for maintaining the geometric structure of the DHT-based overlay. The maintenance cost is affected primarily by the failure detection and failure recovery mechanisms. We evaluated the effects of maintenance overhead on scoped broadcast via extensive simulations. The evaluation shows that it is important to exploit fresh nodes as the neighbors of a node. In addition, cooperative failure discovery and recovery can efficiently and effectively disseminate scoped broadcast messages.

One strategy to tackle the complexity and scalability issue in large-scale mobile ad hoc networks (MANETs) is to use extra layers of abstraction. A common tactic is to group the nodes in the network into clusters. The clusters and the paths between them constitute an extra layer of overlay abstraction. To maintain the overlay structure, a head node is often elected in each cluster. The head nodes form a backbone to provide, among other things, passages from one cluster to another. Nonetheless, the use of dedicated head nodes also creates problems such as load and power imbalance. In this paper, we investigate the feasibility of building a cluster-based overlay mesh on MANETs without using cluster heads. Without head nodes, the challenge is in maintaining the overlay structure and performing inter-cluster routing. We will examine one possible scheme through simulation.

Summary form only given. In mobile agent research, agent-to-agent (A2A) communication often relies on an infrastructure of the discovery server to resolve the location of the agents. This not only induces overhead on managing, maintaining and deploying these discovery servers, but also complicates the development of agent applications. While considering scalability and reliability, such an agent discovery infrastructure becomes very sophisticated. We propose a novel A2A communication system, called Armada, which is autonomic and it does not require a discovery infrastructure. Armada is scalable and reliable, and suitable for a wide-area environment with dynamic entities. It does not pose mobile constrains to the agents. Each agent can be globally and uniquely identified without regarding its location. We evaluate Armada through large-scale simulation and prototype measurement in a cluster environment with 35-node desktop PCs. Results from both accesses presents an efficient design of Armada.

Peer-to-Peer (P2P) systems are emerging Internet applications that exploit the resources distributed in the network edges. The resources are aggregated through ad hoc communication among the participating peers, instead of being provided from the centralized servers. The distributed nature of P2P systems avoids performance bottleneck, single points of failure, and censorship of centralized servers. P2P systems also accommodate dynamics in the network and exploit unused resources. P2P systems can generally be classified into unstructured and structured systems. In this paper, we focus on structured P2P systems and review some well-known systems. Their essential features are abstracted, which leads to a tree model to characterize a major family of structured P2P systems.

Peer-to-peer (P2P) overlays are appealing, since they can aggregate resources of end systems without relying on sophisticated infrastructures. Services can thus be rapidly deployed over such overlays. Primitive P2P overlays only support searches with single keywords. For queries with multiple keywords, presently only unstructured P2P systems can support by extensively employing message flooding. We propose a similarity information retrieval system called Meteorograph for structured P2P overlays without relying on message flooding. Meteorograph is fault-resilient, scalable, responsive and self-administrative, which is particularly suitable for an environment with an explosion of information and a large number of dynamic entities. An information item stored in Meteorograph is represented as a vector. A small angle between two vectors means that the corresponding items are characterized by some identical keywords. Meteorograph further stores similar items at nearby locations in the P2P overlay. To retrieve similar items, only nodes in nearby locations are located and consulted. Meteorograph is evaluated with simulation. The results show that Meteorograph can effectively distribute loads to the nodes. Discovering a single item and a set (in size k) of similar items takes O(log N) and (k/c)/spl middot/O(log N) messages and hops respectively, where N is the number of nodes in the overlay and c is the storage capacity of anode.

A wide-area service discovery infrastructure provides a repository in which services over a wide area can register themselves and clients everywhere can inquire about them. We discuss how to build such an infrastructure based on the peer-to-peer model. The proposed system, called Neuron, can be executed on top of a set of federated nodes across the global network and aggregate their resources to provide the discovery service. Neuron is self-organizing, self-tuning, and capable of tolerating failures of nodes and communication links. In addition, it allows the services to be described with arbitrary forms and the system load to be distributed evenly to the nodes. Neuron also supports event notification. We evaluated Neuron via simulation. The preliminary results show that service registration, discovery and service state advertising take at most O(log N) hops to complete.

To support the real-time transport protocol (RTP) over wireless networks is essential for transporting real-time data streams. However, the RTP introduces significant header overheads, which cannot be adequately employed in wireless networks. An enhanced compressed RTP (ECRTP) with mobility support for wireless networks is presented, which is based on the, compressed RTP (CRTP). The ECRTP uses the local-link retransmission mechanism to recover transmission errors over wireless links. The simulation results show that ECRTP consumes insignificant network bandwidth and maximizes successfully received packets compared with RTP and CRTP in an error-prone and bandwidth limited wireless network. Consequently, the playback quality of real-time data streams is improved for wireless networks.

State-of-the-art cellular networks such as General Packet Radio Service systems (GPRS) with enhanced data rates for global evolution (EDGE) air interfaces can support a wide variety of applications, e.g., Web browsing and video conferencing. The desired quality of communication channels, however, for such wireless networks depends on the interference contributed by location errors and elapsed signal strength. To accommodate different quality-of-service (QoS) classes for such widely received applications, an efficient scheduling for air interfaces must essentially be supported. We propose a dynamic scheduling algorithm by together taking the interference and varying QoS requirements into consideration for GPRS with EDGE air interfaces. Conceptually, the proposed scheme assigns priority in order to the conversational, streaming, interactive and background traffic classes defined by ETSI. Each channel is equipped with an A-buffer to gather the acknowledge messages of the successfully transmitted packets to measure the utilization of the associated channel. A global monitoring device tracks each mobile station's state. Based on the measured channel utilization and the states of mobile stations, the proposed scheduling dynamically assigns channels to the ongoing traffic. From the event-driven simulations, the scheme can significantly outperform the scheme without any support of dynamic channel assignment, especially for the interactive and background traffic classes.

The third-generation wideband CDMA (W-CDMA) systems support the relative higher and variable bit rate transmissions for applications with various quality-of-service (QoS) requirements. Code assignment and reassignment schemes in W-CDMA are essential, with the aim of boosting utilization of codes. A dynamic code assignment scheme (DCA) is proposed for traffic with various QoS classes introduced by Universal Mobile Telecommunication System (UMTS). From the simulation results, the proposed DCA can reduce the new call blocking probability and thus improve the system utilization for W-CDMA.

Directory hints help a node in a NOW-based shared memory multiprocessor to keep track of where valid copies of a memory block may reside. With this information the node can fetch the block directly from those nodes on a read miss. In this way the number of network transactions to serve the miss may be reduced and the expensive directory lookup operation may be removed from the critical path. We discuss the issues involved in implementing the directory hint scheme on a NOW-based shared memory multiprocessor and examine one such implementation, which employs a small and fast cache to store the hints. Our simulation results show that the directory hint scheme can effectively reduce the read stall time. Also its performance is very competitive compared with a more expensive implementation which uses a large level-three cache. A drawback of the scheme is that it will introduce extra network traffic. We believe that the state-of-the-art interconnection networks, such as those built upon the SCI Spider (Galles, 1996) and the Intel Cavallino (Carbonaro and Verhoorn, 1996) chips, provide the opportunity to make the directory hint scheme feasible even with the slower network such as the one built by Myrinet switches (N.J.B et al., 1996).

In distributed shared memory (DSM) multiprocessors, a write operation requires multiple messages to invalidate the nodes which share and cache the memory block to be written. The resulting write stall time is a performance hurdle to such systems. One approach to efficient invalidation is to use multicast messages to reach the sharing nodes. We use application driven simulation to evaluate two multicast based invalidation schemes: dual path (X. Lin and L.M. Ni, 1993) and pruning (M.P. Malumbres et al., 1996). Based on our experimental settings, we found that multicast improves invalidation traffic for four of the six evaluated real applications. The remaining two programs are computation intensive, and multicast based validation is less effective. But since they induce bursty communication, we found that multicasts help to relieve the network congestion during those periods of time. Dual path performs a little better than pruning, because it is less sensitive to routing delay in the routers. We also found that cache size is an important design parameter for multicast based invalidation. It is more effective for DSM multiprocessors with large caches.

With the vast advances of Internet services, large-scale and high-performance servers, such as CC-NUMA multiprocessors, are gaining importance in network computing. In a CC-NUMA multiprocessor, the key component to connect a computing node to the interconnection network is the node controller. Node controllers perform protocol processing to transmit messages with other nodes in the system. As the new generation CC-NUMA multiprocessors are moving towards application-specific protocol processing, a node controller will require very powerful protocol processors or engines to provide the flexibility of processing different kinds of protocols. In this paper, we study the design of a thread-based node controller, in which protocol engines have a multithreaded architecture. Multithreading allows protocol processing of different requests to proceed in parallel, whereby reducing blocking and improving response time. Four important design parameters for a multithreaded protocol engine are examined: (1) the number of thread context storages, (2) the number of protocol operation units, (3) the scheduling policy and (4) the thread allocation scheme. From the application-driven simulation on six representative applications, we conclude that the number of thread contexts and protocol operation units have a great impact on the overall system performance. An appropriate thread allocation scheme for invalidation traffic is needed, and prioritizing a thread and scheduling it accordingly are also important for the system performance.

Cache depot is a performance enhancement technique on cache-coherent non-uniform memory access (CC-NUMA) multiprocessors, in which nodes in the system store extra memory blocks on behalf of other nodes. In this way memory requests from a node can be satisfied by nearby depot nodes without going all the way to the home node. This not only reduces memory access latency and network traffic, but also spreads the network load more evenly. We study the design strategy for cache depot that: enhances the network interface of each node to include a depot cache, which stores those extra memory blocks for other nodes; and employs a new multicast routing scheme, which is called the multi-hop worms and works cooperatively with depot caches, to transmit coherence messages. By considering message routing and depot caches together the design concept can be applied even to those CC-NUMA systems that have a non-hierarchical, scalable interconnection network. We have developed an execution-driven simulator to evaluate the effectiveness of the design strategy. Performance results from using four SPLASH-2 benchmarks show that the design strategy improves the performance of the CC-NUMA multiprocessor by 11% to 21%. We have also studied in depth various factors which affect the performance of cache depot.