We present in this paper two novel infrastructural services based on Hadoop for big data storage and computing in a Taiwan's semiconductor wafer fabrication foundry. The two services include Hadoop data service (HDS) and distributed R language computing service (DRS), which have been built and operated in production systems for 3.5 years. They evolve over time by incrementally accommodating users' requirements. HDS is a web- based distributed big data storage facility. Users simply rely on HDS to access data objects stored in Hadoop with the HTTP protocol. In addition, HDS is scalable and reliable. Moreover, HDS is efficient and effective by intelligently selecting either Hadoop distributed file system (HDFS) or database (HBase) for publishing data objects. Specifically, HDS is transparent to existing analytics and data inquiry applications, such as Spark and Hive. While HDS is a unified storage for supporting sequential and random data accesses for big data in the wafer fabrication foundry, DRS is a distributed computing framework for typical R language users. R users employ DRS to enjoy data-parallel computations, effortlessly and seamlessly. Similar to HDS, DRS can be horizontally scaled out. It guarantees the completion of computational jobs even with failures. In particular, it adaptively reallocates computational resources on the fly, minimizing job execution time and maximizing utilization of allocated resources. This paper discusses the design and implementation features for HDS and DRS. It also demonstrates their performance metrics.

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

Distributed file systems (DFS) are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions, a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance, that is, the file chunks are not distributed as uniformly as possible in the nodes. Although distributed load balancing algorithms exist in the literature to deal with the load imbalance problem, emerging DFSs in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, we illustrate and define the load rebalancing problem in cloud DFSs. We advocate file systems in clouds shall incorporate decentralized load rebalancing algorithms to eliminate the performance bottleneck and the single point of failure. Simulation results for a potential distributed load balancing algorithm are illustrated. The performance of our proposal implemented in the Hadoop distributed file system is also demonstrated.

SUMMARY The rapid progress of information techniques enables people to have a relatively convenient environment and life. This study designs and develops the Web2.0‐based Home Appliances controL sErvice platform (WHALE) for users to easily control home appliances. WHALE is a three‐tier architecture, which consists of Device Detection and Registration Service Platform (DDRSP), Web2.0 Service Platform (WSP), and Common Service Platform (CSP). WHALE is based on the standards of Devices Profile for Web Services (DPWS), Open Services Gateway initiative (OSGi), and Web2.0 techniques. Users can use many kinds of handheld devices to get services from WSP. When DDRSP finds a new home appliance, this home appliance will be registered into CSP. CSP records the related information and services which are provided by this home appliance. WSP not only integrates and shows the related services of all home appliances but also provides controlling interfaces to control the actions of all home appliances. All delivered messages are based on Simple Object Access Protocol (SOAP) and Web Services Description Language (WSDL). Furthermore, for the Web QoS (Quality of Service) strategy, this study designs and proposes an Intelligent Web qualIty of Service‐aware maTching (iWIST) model to satisfy different characteristics of the user's requirements. iWIST consists of Components Classifier, Admission Controller, and Dynamic Content Adaptor. Component Classifier can classify users' requesting components with different QoS constraints. Admission Controller determines the maximum available components for the user. Dynamic Content Adaptor dynamically delivers component information to users. Experiment results illustrate that WHALE can provide appropriate contents with low response time to users. Finally, WHALE has integrity, scalability, operability, adaptivity, and extensibility functions. It can be a reference model for developing home appliance controllers in the future. Copyright © 2011 John Wiley & Sons, Ltd.

The demand for a faster, more efficient and more convenient way for accessing the WWW is urgently required with the growing use of the Internet. Thus, even for many non-computer devices, such as telephones, fax machines and hand-held devices, there have been added some new networking features and corresponding new applications created. Different I/O devices have different characteristics and browsing behaviors. To provide a versatile service to different I/O devices with different quality of information, we propose a mark-up language called the Quality of Information Mark-up Language (QIML( and corresponding browsing systems in this paper. With QIML, a page designer can describe a different quality of information for presentations to different I/O devices using a single Web page set. The language structure, system architecture and system development are presented in the paper.