Model-based reinforcement learning (MBRL) has been proposed as a promising alternative solution to tackle the high sampling cost challenge in the canonical RL, by leveraging a system dynamics model to generate synthetic data for policy training purpose. The MBRL framework, nevertheless, is inherently limited by the convoluted process of jointly optimizing control policy, learning system dynamics, and sampling data from two sources controlled by complicated hyperparameters. As such, the training process involves overwhelmingly manual tuning and is prohibitively costly. In this research, we propose a "reinforcement on reinforcement" (RoR) architecture to decompose the convoluted tasks into two decoupled layers of RL. The inner layer is the canonical MBRL training process which is formulated as a Markov decision process, called training process environment (TPE). The outer layer serves as an RL agent, called intelligent trainer, to learn an optimal hyperparameter configuration for the inner TPE. This decomposition approach provides much-needed flexibility to implement different trainer designs, referred to "train the trainer." In our research, we propose and optimize two alternative trainer designs: 1) an unihead trainer and 2) a multihead trainer. Our proposed RoR framework is evaluated for five tasks in the OpenAI gym. Compared with three other baseline methods, our proposed intelligent trainer methods have a competitive performance in autotuning capability, with up to 56% expected sampling cost saving without knowing the best parameter configurations in advance. The proposed trainer framework can be easily extended to tasks that require costly hyperparameter tuning.

MLModelCI provides multimedia researchers and developers with a one-stop platform for efficient machine learning (ML) services. The system leverages DevOps techniques to optimize, test, and manage models. It also containerizes and deploys these optimized and validated models as cloud services (MLaaS). In its essence, MLModelCI serves as a housekeeper to help users publish models. The models are first automatically converted to optimized formats for production purpose and then profiled under different settings (e.g., batch size and hardware). The profiling information can be used as guidelines for balancing the trade-off between performance and cost of MLaaS. Finally, the system dockerizes the models for ease of deployment to cloud environments. A key feature of MLModelCI is the implementation of a controller, which allows elastic evaluation which only utilizes idle workers while maintaining online service quality. Our system bridges the gap between current ML training and serving systems and thus free developers from manual and tedious work often associated with service deployment. We release the platform as an open-source project on GitHub under Apache 2.0 license, with the aim that it will facilitate and streamline more large-scale ML applications and research projects.

Data center (DC) plays an important role to support services, such as e-commerce and cloud computing. The resulting energy consumption from this growing market has drawn significant attention, and noticeably almost half of the energy cost is used to cool the DC to a particular temperature. It is thus an critical operational challenge to curb the cooling energy cost without sacrificing the thermal safety of a DC. The existing solutions typically follow a two-step approach, in which the system is first modeled based on expert knowledge and, thus, the operational actions are determined with heuristics and/or best practices. These approaches are often hard to generalize and might result in suboptimal performances due to intrinsic model errors for large-scale systems. In this paper, we propose optimizing the DC cooling control via the emerging deep reinforcement learning (DRL) framework. Compared to the existing approaches, our solution lends itself an end-to-end cooling control algorithm (CCA) via an off-policy offline version of the deep deterministic policy gradient (DDPG) algorithm, in which an evaluation network is trained to predict the DC energy cost along with resulting cooling effects, and a policy network is trained to gauge optimized control settings. Moreover, we introduce a de-underestimation (DUE) validation mechanism for the critic network to reduce the potential underestimation of the risk caused by neural approximation. Our proposed algorithm is evaluated on an EnergyPlus simulation platform and on a real data trace collected from the National Super Computing Centre (NSCC) of Singapore. The resulting numerical results show that the proposed CCA can achieve up to 11% cooling cost reduction on the simulation platform compared with a manually configured baseline control algorithm. In the trace-based study of conservative nature, the proposed algorithm can achieve about 15% cooling energy savings on the NSCC data trace. Our pioneering approach can shed new light on the application of DRL to optimize and automate DC operations and management, potentially revolutionizing digital infrastructure management with intelligence.

Recently, many models have been developed to predict video Quality of Experience (QoE), yet the applicability of these models still faces significant challenges. Firstly, many models rely on features that are unique to a specific dataset and thus lack the capability to generalize. Due to the intricate interactions among these features, a unified representation that is independent of datasets with different modalities is needed. Secondly, existing models often lack the configurability to perform both classification and regression tasks. Thirdly, the sample size of the available datasets to develop these models is often very small, and the impact of limited data on the performance of QoE models has not been adequately addressed. To address these issues, in this work we develop a novel and end-to-end framework termed as DeepQoE. The proposed framework first uses a combination of deep learning techniques, such as word embedding and 3D convolutional neural network (C3D), to extract generalized features. Next, these features are combined and fed into a neural network for representation learning. A learned representation will then serve as input for classification or regression tasks. We evaluate the performance of DeepQoE with three datasets. The results show that for small datasets (e.g., WHU-MVQoE2016 and Live-Netflix Video Database), the performance of state-of-the-art machine learning algorithms is greatly improved by using the QoE representation from DeepQoE (e.g., 35.71% to 44.82%); while for the large dataset (e.g., VideoSet), our DeepQoE framework achieves significant performance improvement in comparison to the best baseline method (90.94% vs. 82.84%). In addition to the much improved performance, DeepQoE has the flexibility to fit different datasets, to learn QoE representation, and to perform both classification and regression problems. We also develop a DeepQoE based adaptive bitrate streaming (ABR) system to verify that our framework can be easily applied to multimedia communication service. The software package of the DeepQoE framework has been released to facilitate the current research on QoE.

Motivated by the prowess of deep learning (DL) based techniques in prediction, generalization, and representation learning, we develop a novel framework called DeepQoE to predict video quality of experience (QoE). The end-to-end framework first uses a combination of DL techniques (e.g., word embeddings) to extract generalized features. Next, these features are combined and fed into a neural network for representation learning. Such representations serve as inputs for classification or regression tasks. Evaluating the performance of DeepQoE with two datasets, we show that for the small dataset, the accuracy of all shallow learning algorithms is improved by using the representation derived from DeepQoE. For the large dataset, our DeepQoE framework achieves significant performance improvement in comparison to the best baseline method (90.94% vs. 82.84%). Moreover, DeepQoE, also released as an open source tool, provides video QoE research much-needed flexibility in fitting different datasets, extracting generalized features, and learning representations.

For decades, ASHRAE Standard 55 has been using the Fanger's predicted mean vote (PMV) model to evaluate the indoor thermal comfort satisfaction. However, this canonical model has drawbacks in both data inadequacy and lack of inputs from test subjects. In this paper, we propose a learning-based solution for thermal comfort modeling via the emerging machine learning techniques and Internet of Things-based pervasive sensing technologies. First, we build an intelligent thermal comfort management (iTCM) system. It adopts the wireless sensor network to collect environmental data and utilizes the wearable device for vital sign monitoring. In addition, a cloud-based back-end system, with cost efficient deployment fees, is developed for data management and analysis. Second, we implement a black-box neural network (NN), namely the intelligent thermal comfort NN (ITCNN). To evaluate the performance of ITCNN, we compare it with the PMV model, three traditional white-box machine learning approaches and three classical black-box machine learning methods. Our preliminary results show that four black-box methods achieve better performance than the PMV model and the three white-box approaches. The ITCNN achieves the best performance and outperforms the PMV model by on average 13.1% and up to 17.8%. Third, with the iTCM system, we demonstrate a novel deep reinforcement learning-based application by encouraging human behavioral changes to form energy-saving habits for greener, smarter, and healthier building. Finally, we discuss the limitations of this paper and present the plan for our future research.

Cloud-centric media network (CCMN) was previously proposed to provide cost-effective content distribution services for user-generated contents (UGCs) based on media cloud. CCMN service providers orchestrate cloud resources to deliver UGCs in a pay-per-use style, with an objective to minimize the operational monetary cost. The monetary cost depends on the actual usage of cloud resources (e.g., computing, storage, and bandwidth), which in turn, is affected by the content placement strategy. In this paper, we investigate this cost-optimal content placement problem. Specifically, it is formulated into a constrained optimization problem, in which the objective is to minimize the total monetary cost, with respect to the resource capacity. We tackle this problem via a two-step strategy. The first step focuses on the placement for a single content, which is mapped into a k-center problem. Using a graph-theoretic approach, we derive and verify a logarithmic model between the optimal mean hop distance from viewers to contents, and the optimal number of content replicas. The second step leverages this analytical result to solve the cost optimization problem, via a feasible direction method. The analysis is substantiated via numerical simulations, using a set of data traces from a top content website. This investigation suggests that the optimal number of content replica for each title follows a power-law distribution in respect to its popularity rank. Moreover, it reveals a fundamental tradeoff between the storage and bandwidth cost. Finally, compared to existing heuristics, our proposed algorithm is able to obtain the optimal placement strategy, with lower computational complexity.

In this paper, we examine the information-theoretic security of multiple-input-multiple-output space-division multiplexed (MIMO-SDM) fiber-optic communication systems in the presence of multiple eavesdroppers. In particular, we analyze the achievable secrecy rate for the two special cases that reflect different capabilities of the eavesdroppers: independent eavesdroppers who do not share received information and colluding (cooperating) eavesdroppers who can combine and jointly process the received signals. Our results show that MIMO-SDM systems are robust against multiple independent tapping attacks, in the sense that the average achievable secrecy rate is strictly positive even with an infinite number of eavesdroppers. Though extremely difficult to implement in practice, if all the eavesdroppers could cooperate coherently, the average secrecy rate decreases quickly with the number of eavesdroppers. As such, to counter the colluding eavesdroppers, a combination of much higher SNR for the legitimate receiver and higher mode-dependent loss (MDL) for the eavesdroppers is needed.

Space-division multiplexed (SDM) fiber optic transmission systems can not only increase system capacity, but also achieve physical-layer security against fiber tapping attacks. In this paper, we examine the information-theoretic security of optical multiple-input-multiple-output (MIMO) SDM by evaluating the tradeoff between the achievable information rate and the confidentiality for different channel dynamics. In particular, we provide problem formulations for secure communication over these channels and study three types of secrecy capacities: 1) guaranteed capacity; 2) outage capacity; and 3) average capacity, each serving as a performance metric for a coding strategy tailored to a specific type of MIMO-SDM channel. We also assess the impact of key system parameters, such as the number of modes, the mode-dependent loss (MDL), and the signal-to-noise ratio (SNR), on the various secrecy capacities. Our results indicate that, with a proper design of channel codes that balance information rate and security, an SDM system has the potential of offering confidential data transmission at a rate that could be orders of magnitude higher than what can be achieved through other means of encryption. Moreover, we show that MDL, unavoidably induced by fiber tapping, can allow information-theoretic security even if the SNR of the eavesdropper's receiver is better than that of the legitimate receiver.

Optical networks are vulnerable to physical layer attacks; wiretappers can improperly receive messages intended for legitimate recipients. Multimode fiber (MMF) transmission can be modeled via a broadcast channel in which both the legitimate receiver's and wiretapper's channels are multiple-input-multiple-output complex Gaussian channels. This work considers the theoretical aspects of this security problem in the domain of a broadcast channel. Source-channel coding analyses based on the use of distortion as the metric for secrecy are developed. Alice has a source sequence to be encoded and transmitted over this broadcast channel so that the legitimate user Bob can reliably decode it while forcing the distortion of the wiretapper, or eavesdropper, Eve's estimate to be as high as possible. Tradeoffs between transmission rate and distortion under two extreme scenarios are examined: the best case where Eve has only her channel output and the worst case where she also knows the past realization of the source. It is shown that under the best case, an operationally separate source-channel coding scheme guarantees maximum distortion at the same rate as needed for reliable transmission. Theoretical bounds are given, and particularized for MMF. Numerical results showing the rate distortion tradeoff are presented and compared with corresponding results for the perfect secrecy case.

In this paper, we study the efficacy of error control schemes for energy-efficient reliable delivery of large files (hundreds of GBs) over core optical networks. Specifically, we examine two schemes: automatic repeat request (ARQ), and hybrid ARQ (i.e. ARQ combined with forward error correction (FEC) capability). We focus on Reed-Solomon (RS) FEC codes (in hybrid ARQ) and propose a new model, incorporating different block sizes as well as code error-correction capability, to estimate the energy consumption for performing encoding and decoding operations in optical networks. The model considers the impact of varying pre-FEC bit-error rates (BER) of the optical channel, and the signal processing blocks used to implement RS codes. Our results show that when the pre-FEC channel BER is in excess of 10-5, hybrid ARQ offers better performance than ARQ in terms of energy efficiency. However, both hybrid ARQ and ARQ have similar performance under lower BER.

Consider a content delivery network in which storage and transport resources, characterized by their capacity and cost (e.g., energy) efficiency, are used to meet users' content object requests. The goal is to find the evolution of the objects being stored and transported by the network resources that meets user requests, satisfies network resource capacities and minimizes overall network cost. We first present a constructive offline solution that provides the maximum network efficiency (or minimum cost per object delivered) that can be achieved by dynamically exploiting network-coded caching and multicasting under arbitrary time-varying demands. We refer to the solution scheme as a dynamic network-coded caching-aided multicast (NCCAM) scheme, and illustrate it in a 6-node butterfly network. We then consider a single time period in which each user requests an arbitrary subset of content objects. We formulate the problem as a network coding problem on a caching-augmented graph and show that under uniform demand, random linear coded caching and multicasting is sufficient for achieving minimum cost caching-aided multicast. For the arbitrary demand scenario, we provide the transport-storage-popularity tradeoff of a polynomial-time solution that uses uncoded caching according to object popularity and random linear coded transmission. We show that while for skewed Zipf object popularity such a simple scheme achieves close to optimal performance, as the Zipf parameter approaches zero (uniform popularity), significant cost reductions can be obtained by optimizing the transport configuration at the expense of increased computational complexity.

In recent years, technical challenges are emerging on how to efficiently distribute the rapid growing user-generated contents (UGCs) with long-tailed nature. To address this issue, we have previously proposed cloud centric media network (CCMN) for cost-efficient UGCs delivery. In this paper, we further study the content placement problem in CCMN. Our objective is to minimize the monetary cost incurred by using cloud resources to orchestrate an elastic and global content delivery network (CDN) service. In particular, this objective is achieved via a two-step method. First, for a single content, we map it into a k-center problem, and find a logarithmic relationship between the mean hop distance from users to contents, and the reciprocal of replica number. Second, for multiple contents, we formulate a convex optimization with storage and bandwidth capacity constraints, which can be solved by our proposed algorithm. Finally, we verify the algorithm based on real-world traces collected from a popular video website in China. Our numerical results suggest that, the optimal number of replica for each content follows a power law in respect to its popularity, under feasible storage and bandwidth constraints, in a set of deployed backbone networks.

Consider a network of prosumers of media content in which users dynamically create and request content objects. The request process is governed by the objects' popularity and varies across network regions and over time. In order to meet user requests, content objects can be stored and transported over the network, characterized by the capacity and energy efficiency of the storage and transport resources. The energy efficient dynamic in-network caching problem aims at finding the evolution of the network configuration, in terms of the content objects being cached and transported over each network element at any given time, that meets user requests, satisfies network resource capacities and minimizes overall energy use. We provide 1) an information-centric optimization framework for the energy efficient dynamic in-network caching problem, 2) an offline solution, EE-OFD, based on an integer linear program (ILP) that obtains the maximum efficiency gains that can be achieved with global knowledge of user requests and network resources, and 3) an efficient fully distributed online solution, EEOND, that allows network nodes to make local caching decisions based on their current estimate of the global energy benefit. Our solutions take into account the network heterogeneity, in terms of capacity, energy efficiency and content popularity, and adapt to changing network conditions minimizing overall energy use.

This paper provides a theoretical framework of energy-optimal mobile cloud computing under stochastic wireless channel. Our objective is to conserve energy for the mobile device, by optimally executing mobile applications in the mobile device (i.e., mobile execution) or offloading to the cloud (i.e., cloud execution). One can, in the former case sequentially reconfigure the CPU frequency; or in the latter case dynamically vary the data transmission rate to the cloud, in response to the stochastic channel condition. We formulate both scheduling problems as constrained optimization problems, and obtain closed-form solutions for optimal scheduling policies. Furthermore, for the energy-optimal execution strategy of applications with small output data (e.g., CloudAV), we derive a threshold policy, which states that the data consumption rate, defined as the ratio between the data size (L) and the delay constraint (T), is compared to a threshold which depends on both the energy consumption model and the wireless channel model. Finally, numerical results suggest that a significant amount of energy can be saved for the mobile device by optimally offloading mobile applications to the cloud in some cases. Our theoretical framework and numerical investigations will shed lights on system implementation of mobile cloud computing under stochastic wireless channel.

In content-centric networking (CCN), the in-network caching feature provides several attractive advantages such as low dissemination latency and network transport load reduction. Thus, CCN requires less transport energy but additional energy to provide a caching capability at every content router. In this paper, we investigate the minimum energy consumption that CCN can achieve with optimal cache locations by considering different caching hardware technologies, number of downloads per hour, and content popularity. We first set up an energy consumption model for CCN and then formulate linear and nonlinear programming problems to minimize total energy consumption of CCN. Also, a genetic algorithm (GA) approach is proposed to find energy-efficient cache locations. Using reported energy efficiency of computational hardware and network equipment, we show CCN yield greater energy savings for very popular content and small-sized catalog, compared to conventional CDN. Our results also indicate that two aspects of the memory technology, energy-proportional caching and sufficient memory capacity, are critical to the overall energy efficiency gain of CCN.

In-network processing of media streams will be necessary in order to meet the personalization, interactive, and real time requirements of future video centric media services. Using multi-view video (MVV) streaming as an example, we investigate the energy tradeoff between video processing and transport for the delivery of personalized media services. We focus on evaluating the energy benefit of distributed vs. centralized processing architectures. We provide solutions for the relative energy efficiency regions as a function of the user viewing preferences and the processing-transport efficiency ratio. Our results show that a small number of requests and a homogeneous interest among viewing regions favors the centralized processing of personalized video streams and multicast transport to end users, while a larger number of requests and a heterogeneous interest favors the processing of personalized views at a distributed subset of nodes in the network.

We investigate the potential trade-off between energy efficiency and latency of file transmission over IP and Optical domains. We consider various queueing disciplines that are applicable to these domains, and show that existing schemes achieve only one at the expense of the other. We show that we can simultaneously achieve low-latency file transfers in an energy-efficient manner using an intuitive approach that is easy to implement in next-generation networks.

We examine the basic energy tradeoffs between video transport and video processing for services such as multi-view video (MVV) streaming, where multiple media streams are combined and processed to create an immersive and personalized user experience. We analyze and compare the energy efficiency of different architectural options for the location of video processing functions and illustrate how the architecture of choice is influenced by the network topology, the users' view preferences, and the relative transport-processing energy efficiency. We provide an integer linear programming formulation for the energy efficient functional resource allocation problem, which we show it can be solved as a linear program, and an easily implementable algorithm that generates optimal solutions in polynomial time.

In this paper, we examine how energy constraints might shape future optical communication networks and the impact that current technology trends may have on future energy use. Historical factors and prevailing complications associated with fiber capacity point to an increased focus on energy to enable tighter photonic and electronic component integration and larger networks. Energy requirements and associated challenges are described at the component, system, and network level.

The challenges of optimal flooding of network control messages in mobile ad-hoc network (MANET) have been well studied in the literature [2]-[10] and [12]-[19]. A particular case of this problem is encountered in MANET routing protocols, which need to distribute routing control information to all member nodes as efficiently as possible. While both proactive and reactive MANET routing protocols exist, in some environments a proactive approach will be used to reduce communications latency as much as possible. Even in a hybrid protocol, the proactive portion requires some amount of information exchange among all nodes. Many previous papers have discussed optimal flooding in a variety of environments under a variety of assumptions, but none have assessed the performance of a recently developed cluster based routing protocol called Adaptive hybrid domain routing (AHDR). In this paper, the optimal flooding architecture developed for the clustered based AHDR protocol is discussed in detailed. The paper also compares AHDR's flooding architecture against two prominent MANET routing protocols, OLSR [3] and OSPF- MDR [11]. The paper also shows simulation results for each of the flooding architectures using an unbiased and non-intrusive monitoring technique. The performance results show that AHDR's flooding architecture imposes lower overhead control traffic as compared to OLSR and OSPF-MANET-MDR while still providing better network reachability.

In this paper, we provide a parametric model for analyzing the control overhead usage In mobile ad-hoc networks (MANET). In particular, we focus on evaluating the Impact of the hierarchy of both topology formation and routing on the scalability of overhead usage. Via an analytical approach, we quantitatively characterize the amount of overhead under different domain sizes, as well as other design parameters. The results help us to assess optimal scaling of overhead traffic as a function of network size N and to provide insights into developing scalable routing architectures for MANET.