Abstract Model monitoring is a critical component of the machine learning lifecycle, safeguarding against undetected drops in the model’s performance after deployment. Traditionally, performance monitoring has required access to ground truth labels, which are not always readily available. This can result in unacceptable latency or render performance monitoring altogether impossible. Recently, methods designed to estimate the accuracy of classifier models without access to labels have shown promising results. However, there are various other metrics that might be more suitable for assessing model performance in many cases. Until now, none of these important metrics has received similar interest from the scientific community. In this work, we address this gap by presenting Confidence-based Performance estimation (CBPE), a novel method that can estimate any binary classification metric defined using the confusion matrix. In particular, we choose four metrics from this large family: accuracy, precision, recall, and $$\hbox {F}_1$$ , to demonstrate our method. CBPE treats the elements of the confusion matrix as random variables and leverages calibrated confidence scores of the model to estimate their distributions. The desired metric is then also treated as a random variable, whose full probability distribution can be derived from the estimated confusion matrix. CBPE is shown to produce estimates that come with strong theoretical guarantees and valid confidence intervals.

ABSTRACT Introduction As machine learning (ML)‐enabled systems become increasingly prevalent across industries, the engineering challenges of deploying and maintaining them in production have emerged as critical. The existing engineering knowledge base often derives from conceptual frameworks or case studies conducted by large technology companies, leaving a gap in the empirical nuances of deployment practices across a wide range of engineering contexts. Objective This study aims to investigate real‐world ML deployment workflows and inference architectures to identify common patterns, contextual variations, and underlying trade‐offs that influence how practitioners operationalize ML models in production. Methods We conducted a multi‐case study of eight ML systems across sectors, including advertising, finance, healthcare, manufacturing, and software platforms. We collected data through semi‐structured interviews with practitioners and the various inference architectures. We applied thematic analysis to extract recurring patterns across the deployment lifecycle, from model versioning to inference serving. Results Our findings reveal five core deployment themes: model versioning and storage, quality assurance, monitoring, model packaging, and inference serving. We identify a maturity gradient from manual, ad hoc practices to automated CI/CD pipelines and further highlight architectural trade‐offs between tightly coupled vs loosely coupled model deployment patterns. Other key findings include the rarity of model drift monitoring and the use of hybrid inference serving patterns to balance latency and scalability requirements. Conclusions Although ML domains and deployment architectures vary, we observed recurrent practices and strategic trade‐offs that provide an empirical foundation for developing more standardized, context‐aware ML engineering processes. Our findings provide actionable guidance for practitioners and identify under‐explored areas for future research.

Site reliability engineers manage increasingly complex systems and integrations, often under formal service level agreements (SLAs). A key challenge is balancing alerting strategies to detect service level objective (SLO) breaches-ensuring timely incident response while avoiding overly aggressive rules that lead to alert fatigue. Artificial Intelligence for IT Operations (AIOps) is a research domain focused on enhancing the maintainability and availability of IT systems using machine learning. In this study, we conducted a literature review to define the problem, assess state-of-the-art research in AIOps, and explore advancements in time-critical time series analysis. This research outlines key requirements for machine learning-based SLO alerting, identifying real-time failure detection and failure prediction as primary approaches. The length of the observation window, forecast horizon, and training/inference times are crucial parameters for selecting and optimizing solutions. We highlight machine learning techniques that align with these requirements and address known AIOps challenges. Our findings contribute to a better understanding of the problem space and potential solutions, helping researchers and industry practitioners drive innovation in machine learning-based reliability engineering.

After a machine learning model has been deployed into production, its predictive performance needs to be monitored. Ideally, such monitoring can be carried out by comparing the model’s predictions against ground truth labels. For this to be possible, the ground truth labels must be available relatively soon after inference. However, there are many use cases where ground truth labels are available only after a significant delay, or in the worst case, not at all. In such cases, directly monitoring the model’s predictive performance is impossible. Recently, novel methods for estimating the predictive performance of a model when ground truth is unavailable have been developed. Many of these methods leverage model confidence or other uncertainty estimates and are experimentally compared against a naive baseline method, namely Average Confidence (AC), which estimates model accuracy as the average of confidence scores for a given set of predictions. However, until now the theoretical properties of the AC method have not been properly explored. In this paper, we bridge this gap by reviewing the AC method and show that under certain general assumptions, it is an unbiased and consistent estimator of model accuracy. We also augment the AC method by deriving valid confidence intervals for the estimates it produces. These contributions elevate AC from an ad-hoc estimator to a principled one, encouraging its use in practice. We complement our theoretical results with empirical experiments, comparing AC against more complex estimators in a monitoring setting under covariate shift. We conduct our experiments using synthetic datasets, which allow for full control over the nature of the shift. Our experiments with binary classifiers show that the AC method is able to beat other estimators in many cases. However, the comparative quality of the different estimators is found to be heavily case-dependent.

Abstract Log files can help detect and diagnose erroneous software behaviour, but their utility is limited by the ability of users and developers to sift through large amounts of text. Unsupervised machine learning tools have been developed to automatically find anomalies in logs, but they are usually not designed for situations where a large number of log streams or log files, each with its own characteristics, need to be analyzed and their anomaly scores compared. We propose Ladle, an accurate unsupervised anomaly detection and localization method that can simultaneously learn the characteristics of hundreds of log types and determine which log entries are the most anomalous across these log types. Ladle uses a sentence transformer (a large language model) to embed short overlapping segments of log files and compares new, potentially anomalous, log segments against a collection of reference data. The result of the comparison is re-centered by subtracting a baseline score indicating how much variation tends to occur in each log type, making anomaly scores comparable across log types. Ladle is designed to adapt to data drift and is updated by adding new reference data without the need to retrain the sentence transformer. We demonstrate the accuracy of Ladle on a real-world dataset consisting of logs produced by an endpoint protection platform test suite. We also compare Ladle’s performance on the dataset to that of a state-of-the-art method for single-log anomaly detection, showing that the latter is inadequate for the multi-log task.

After a machine learning model has been deployed into production, its predictive performance needs to be monitored. Ideally, such monitoring can be carried out by comparing the model’s predictions against ground truth labels. For this to be possible, the ground truth labels must be available relatively soon after inference. However, there are many use cases where ground truth labels are available only after a significant delay, or in the worst case, not at all. In such cases, directly monitoring the model’s predictive performance is impossible. Recently, novel methods for estimating the predictive performance of a model when ground truth is unavailable have been developed. Many of these methods leverage model confidence or other uncertainty estimates and are experimentally compared against a naive baseline method, namely Average Confidence (AC), which estimates model accuracy as the average of confidence scores for a given set of predictions. However, until now the theoretical properties of the AC method have not been properly explored. In this paper, we bridge this gap by reviewing the AC method and show that under certain general assumptions, it is an unbiased and consistent estimator of model accuracy. We also augment the AC method by deriving valid confidence intervals for the estimates it produces. These contributions elevate AC from an ad-hoc estimator to a principled one, encouraging its use in practice. We complement our theoretical results with empirical experiments, comparing AC against more complex estimators in a monitoring setting under covariate shift. We conduct our experiments using synthetic datasets, which allow for full control over the nature of the shift. Our experiments with binary classifiers show that the AC method is able to beat other estimators in many cases. However, the comparative quality of the different estimators is found to be heavily case-dependent.

Abstract Materials science plays an important role in the field of fusion research. We focus on the binding energy of vacancy-type defects to screw dislocations. These defects produced by irradiation are known to affect the mechanical properties of the material. Traditional techniques, such as density functional theory or molecular dynamics simulations, can be used to study these defects. However, a combinatorial number of cases need to be analyzed to study the binding energy when several vacancies are present, which quickly becomes infeasible. To address this combinatorial issue, we present a neural network solution. From a subset of cases we can train a model, which in turn can predict the energy in a fraction of the time compared to traditional techniques. However, we have to deal with large uncertainties in our predictions. This is addressed by using uncertainty quantification techniques, such as mixture density networks. We present results for a large iron dataset and for a reduced tungsten dataset, in which our solution is shown to benefit from transfer learning. Therefore, we can use the model to analyze different materials while avoiding the cost of generating new large datasets and training the model from scratch. We see a mean absolute percentage error of 7.5% for the iron case and 9.6% for the reduced tungsten case.

Modern machine learning models can achieve impressive results in many tasks, but often fail to express reliably how confident they are with their predictions. In an industrial setting, the end goal is usually not a prediction of a model, but a decision based on that prediction. It is often not sufficient to generate high-accuracy predictions on average. One also needs to estimate the uncertainty and risks involved when making related decisions. Thus, having reliable and calibrated uncertainty estimates is highly useful for any model used in automated decision-making.In this paper, we present a case study, where we propose a novel method to improve the uncertainty estimates of an in-production machine learning model operating in an industrial setting with real-life data. This model is used by Basware, a Finnish software company, to extract information from invoices in the form of machine-readable PDFs. The solution we propose is shown to produce calibrated confidence estimates, which outperform legacy estimates on several relevant metrics, increasing coverage of automated invoices from 65.6% to 73.2% with no increase in error rate.

Computer vision systems can automatically find and segment anomalies in images even without ever seeing anomalous observations during training. Many methods for such unsupervised anomaly detection (AD) and localization (AL) tasks have been introduced in recent years, but the most accurate methods tend to be computationally heavy. In this paper, we propose Discrepancy Scaling, a method that significantly improves the accuracy of a very fast AD and AL approach called Student-Teacher Feature Pyramid Matching. We show that with Discrepancy Scaling, even a small, mobile-friendly convolutional neural network can perform well on AD and AL tasks.

Sound anomaly detection is a task that aims at identifying unusual or abnormal sounds within audio data. These sounds could be caused by different factors, such as background noise, equipment malfunctions, or unexpected events. Anomaly detection in sound is a well-studied topic, with a lot of research being done in the field. Anomaly localization refers to the process of identifying the specific location or region within a sample where an anomaly (or outlier) occurs. When applied to audio signals, anomaly localization can involve analyzing the spectral content of the sound to detect regions that deviate from the typical or expected pattern.In this study, we present a simple yet effective model based on the Student-Teacher Feature Pyramid Matching Method for locating anomalies in audio data. Utilizing the MIMII dataset by augmenting it with synthetic anomalies, we evaluate the method’s accuracy. Our results demonstrate that the proposed model can accurately locate artificially created anomalies within the spectrograms, both in terms of time and frequency. This approach offers a promising solution for identifying and determining the precise location of anomalies in various audio applications.

Machine Learning Operations (MLOps), derived from DevOps, aims to unify the development, deployment, and maintenance of machine learning (ML) models. Continuous training (CT) automatically retrains ML models, and continuous deployment (CD) automatically deploys the retrained models to production. CT and CD are essential for maintaining ML model performance in dynamic production environments and, therefore, need to be considered when practicing MLOps. We present our CTCD-e MLOps pipeline, implemented mostly using existing open-source software, being able to autonomously adapt an ML system to changing production environments by enabling flexible model CT and CD. The pipeline can automatically trigger a model retraining round when the model performance degrades. Then it automatically conducts an A/B test for the retrained model and its predecessor in production to start serving the better one. The pipeline was evaluated by two experiments. In the pipeline, users can flexibly configure the model retraining, as well as the redeployment and production A/B test of the retrained models based on various requirements.

Machine learning (ML) provides us with numerous opportunities, allowing ML systems to adapt to new situations and contexts. At the same time, this adaptability raises uncertainties concerning the run-time product quality or dependability, such as reliability and security, of these systems. Systems can be tested and monitored, but this does not provide protection against faults and failures in adapted ML systems themselves. We studied software designs that aim at introducing fault tolerance in ML systems so that possible problems in ML components of the systems can be avoided. The research was conducted as a case study, and its data was collected through five semi-structured interviews with experienced software architects. We present a conceptualisation of the misbehaviour of ML systems, the perceived role of fault tolerance, and the designs used. Common patterns to incorporating ML components in design in a fault tolerant fashion have started to emerge. ML models are, for example, guarded by monitoring the inputs and their distribution, and enforcing business rules on acceptable outputs. Multiple, specialised ML models are used to adapt to the variations and changes in the surrounding world, and simpler fall-over techniques like default outputs are put in place to have systems up and running in the face of problems. However, the general role of these patterns is not widely acknowledged. This is mainly due to the relative immaturity of using ML as part of a complete software system: the field still lacks established frameworks and practices beyond training to implement, operate, and maintain the software that utilises ML. ML software engineering needs further analysis and development on all fronts.

Sensor data collection in IoT networks is sensitive to malfunction of sensors and communications. Hence, it is important that models using the data work in a reasonable way even when there are some, potentially temporary, problems. In this paper, we investigate the robustness of AutoML systems for time series forecasting in sensor networks, using temperature data as example. We experiment with different AutoML systems and study how the resulting models tolerate faults in their input data. The analyzed AutoML systems are Microsoft's Azure AutoML, Intel's Analytics Zoo AutoML, and Facebook's Prophet. As a result, we rank AutoML systems based on their performance with respect to data faults and their severity. In addition, we show how the AutoML generated models differ given the data fault type.

In this paper, we present a coverage-based regression test selection (RTS) approach and a developed tool for Python. The tool can be used either on a developer’s machine or on build servers. A special characteristic of the tool is the attention to easy integration to continuous integration and deployment. To evaluate the performance of the proposed approach, mutation testing is applied to three open-source projects, and the results of the execution of full test suites are compared to the execution of a set of tests selected by the tool. The missed fault rate of the test selection varies between 0–2% at file-level granularity and 16–24% at line-level granularity. The high missed fault rate at the line-level granularity is related to the selected basic mutation approach and the result could be improved with advanced mutation techniques. Depending on the target optimization metric (time or precision) in DevOps/MLOps process the error rate could be acceptable or further improved by using file-level granularity based test selection.

Artificial intelligence (AI) has made its way into everyday activities, particularly through new techniques such as machine learning (ML). These techniques are implementable with little domain knowledge. This, combined with the difficulty of testing AI systems with traditional methods, has made system trustworthiness a pressing issue. This paper studies the methods used to validate practical AI systems reported in the literature. Our goal is to classify and describe the methods that are used in realistic settings to ensure the dependability of AI systems. A systematic literature review resulted in 90 papers. Systems presented in the papers were analysed based on their domain, task, complexity, and applied validation methods. The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions are methods used to continuously validate the systems after deployment. Our results clarify existing strategies applied to validation. They form a basis for the synthesization, assessment, and refinement of AI system validation in research and guidelines for validating individual systems in practice. While various validation strategies have all been relatively widely applied, only few studies report on continuous validation.

WiFi networks are increasingly subjected to cross-technology interference with emerging IoT and even mobile communication solutions all crowding the 2.4 GHz ISM band where WiFi networks conventionally operate. Due to the diversity of interference sources, maintaining high level of network performance is becoming increasing difficult. Recently, deep learning based interference detection has been proposed as a potentially powerful way to identify sources of interference and to provide feedback on how to mitigate their effects. The performance of such approaches has been shown to be impressive in controlled evaluations. However, little information exists on how they generalize to the complexity of everyday environments. In this paper, we contribute by conducting a comprehensive performance evaluation of deep learning based interference detection. In our evaluation, we consider five orthogonal but complementary metrics: correctness, overfitting, robustness, efficiency, and interpretability. Our results show that, while deep learning indeed has excellent correctness (i.e., detection accuracy), it can be prone to noise in measurements (e.g., struggle when transmission power is dynamically adjusted) and suffers from poor interpretability. Deep learning is also highly sensitive to the quality and quantity of training data, with performance decreasing rapidly when the training and testing measurements come from environments with different characteristics. To compensate for weaknesses of deep learning, as our second contribution we propose a novel signal modeling approach for interference detection and compare it against deep learning. Our results demonstrate that, in terms of errors, there are some differences across the two approaches, with signal modeling being better at identifying technologies that rely on frequency hopping or that have dynamic spectrum signatures but suffering in other cases. Based on our results, we draw guidelines for improving interference detection performance.

Sharing live multimedia content is becoming increasingly popular among mobile users. In this article, we study the problem of optimizing video quality in such a scenario using scalable video coding (SVC) and chunked video content. We consider using only standard stateless HTTP servers that do not need to perform additional processing of the video content. Our key contribution is to provide close to optimal algorithms for scheduling video chunk upload for multiple clients having different viewing delays. Given such a set of clients, the problem is to decide which chunks to upload and in which order to upload them so that the quality-delay tradeoff can be optimally balanced. We show by means of simulations that the proposed algorithms can achieve notably better performance than naive solutions in practical cases. Especially the heuristic-based greedy algorithm is a good candidate for deployment on mobile devices because it is not computationally intensive but it still delivers in most cases on-par video quality compared to the more complex local optimization algorithm. We also show that using shorter video segments and being able to predict bandwidth and video chunk properties improve the delivered video quality in certain cases.

Video streaming can drain a smartphone battery quickly. A large part of the energy consumed goes to wireless communication. In this article, we first study the energy efficiency of different video content delivery strategies used by service providers and identify a number of sources of energy inefficiency. Specifically, we find a fundamental tradeoff in energy waste between prefetching small and large chunks of video content: small chunks are bad because each download causes a fixed tail energy to be spent regardless of the amount of content downloaded, whereas large chunks increase the risk of downloading data that user will never view because of abandoning the video. Hence, the key to optimal strategy lies in the ability to predict when the user might abandon viewing prematurely. We then propose an algorithm called eSchedule that uses viewing statistics to predict viewer behavior and computes an energy optimal download strategy for a given mobile client. The algorithm also includes a mechanism for explicit control of traffic overhead, i.e., unnecessary download of content that the user will never watch. Our evaluation results suggest that the algorithm can cut the energy waste down to less than half compared to other strategies. We also present and experiment with an Android prototype that integrates eSchedule into a YouTube downloader.

Energy efficiency has become a primary concern for data centers in recent years. Understanding where the energy has been spent within a software is fundamental for energy-efficiency study as a whole. In this paper, we take the first step towards this direction by building an energy profiling module on top of IgProf. IgProf is an application profiler developed at CERN for scientific computing workloads. The energy profiling module is based on sampling and obtains energy measurements from the Running Average Power Limit (RAPL) interface present on the latest Intel processors. The initial profiling results of a single-threaded program demonstrates potential, showing a close correlation between the execution time and the energy spent within a function.

New data centers for cloud services rapidly increase energy consumption of IT services. Now electricity is used both to power the computing hardware and to remove the heat computing generates. On the other hand, heating is needed e.g. for buildings and for hot water. However, reusing heat from data centers, e.g. for district heating, is complicated and expensive because of relatively low temperature of exhaust heat and a need for expensive infrastructure investments. In this paper, we will study the feasibility and the technical solutions for distributing computing to where heating is needed. For this purpose compact and reliable computing-heating units redeveloped. Computing tasks are distributed to these units based on the heating requirements in their environment. In this way the electricity used for computing is substituting the energy of the heating elements.

This paper covers a study on WebRTC data channel performance in current web browser implementations. The goal is to find out whether WebRTC data channels are usable today in web applications demanding throughput performance for data transfers consisting of arbitrary data. Performance is measured using a purpose-built web application and various simulated network conditions. Packet capture is used for further analysis. The results reveal that current WebRTC data channel implementations do not adjust the SCTP window size from the default setting. This results in bad performance when network conditions and especially latency is not close to perfect. Changing the window size results in significantly better performance on high latency links, but the observed throughput performance is still not ideal. We can conclude from the test results that current WebRTC data channel implementations are not yet ready for high performance requirements nor mobile environments where battery life is important. The browsers need their WebRTC data channel implementations optimized in order for the technology to become truly useful.

Summary The development of Internet of Things (IoT) applications can be facilitated by encoding the meaning of the data in the messages sent by IoT nodes, but the constrained resources of these nodes challenge the common Semantic Web solutions for doing this. In this article, we examine enabling technologies for adding semantics to the IoT. Especially, we analyze data formats, which enable IoT applications consume semantic IoT data in a straightforward and general fashion, and evaluate resource usage of different alternatives with a sensor system. Our experiment illustrates encoding and decoding of different data formats and shows how big a difference a data format can make in energy consumption. Copyright © 2014 John Wiley & Sons, Ltd.

Recent energy measurements on smartphones have shown that parallel communications (e.g., data transfer and voice call) require less energy than their stand-alone execution. Guided by these results, we investigate the possibility of scheduling communications in pairs for minimizing the energy consumption. We define two energy optimization problems to postpone delay-tolerant services and perform them in parallel with real-time services in order to save energy. The first problem, called single delay-tolerant assignment (SDA), allows at most one delay-tolerant service to be paired with each real-time service, whereas the second problem, called multiple delay-tolerant assignment (MDA), allows multiple delay-tolerant services to be paired (in different times) with the same real-time service. For the SDA problem, we propose an optimal algorithm. For the MDA problem, which is computationally intractable, we give an approximation algorithm. We evaluate the benefits of the energy-efficient pairing strategy via simulations on synthetic traces. The MDA algorithm can save up to the 60 percent of the energy consumption using 4G network assuming an intensive smartphone usage, while the SDA algorithm saves up to the 20 percent.

Urban (and suburban) informal settlements in emerging markets will be the fastest growing population hotspots in the next few decades. This presents a significant challenge in delivery of services, including affordable mobile Internet access. Operator-led upgrades through network densification (rollout of additional operator-maintained sites) are difficult to sustain in those areas due to low revenues, lack of fixed lines, energy scarcity, insecurity, and so on. This calls for alternative approaches in mobile network upgrades and operation. In this paper, we present capacity enhancement approach using user-deployed shared-access small cells in the dense informal settlements. To that end, we consider leveraging macro Long Term Evolution (LTE) networks to backhaul High Speed Packet Access (HSPA) small cells. As a case study, we present comparative network simulations based on an example informal settlement. The results of the study highlight the possibilities for cost-effective capacity upgrades for users in dense settlements for even a limited number of unplanned end-user small deployments and self-backhauling via existing macro sites. In the study, we also note possible system performance enhancements of the small cell backhaul link through improved antenna design, scaling of carrier bandwidth and introduction of traffic steering across HSPA and LTE layers.

This article proposes a novel energy-efficient multimedia delivery system called EStreamer. First, we study the relationship between buffer size at the client, burst-shaped TCP-based multimedia traffic, and energy consumption of wireless network interfaces in smartphones. Based on the study, we design and implement EStreamer for constant bit rate and rate-adaptive streaming. EStreamer can improve battery lifetime by 3x, 1.5x, and 2x while streaming over Wi-Fi, 3G, and 4G, respectively.

Video streaming on smartphones is one of the most popular but also most energy hungry services today. Using mobile video services results in two contradictory sources of energy waste for smartphones: i) energy waste because of excessively aggressive prefetching of content that the user will not watch because of abandoning the session, and ii) excessive amount of tail energy, which is energy wasted by keeping the wireless interface powered on after receiving a chunk of content; this is caused by prefetching chunks that are too small. To remedy this, we propose a novel download scheduling algorithm based on crowd-sourced video viewing statistics. Our algorithm judiciously evaluates the probability of a user interrupting a video viewing in order to perform the right amount of prefetching. In this way, the algorithm balances the amount of the two above-mentioned kinds of energy waste. By simulations, we show that our scheduler cuts the energy waste to half compared to existing download strategies. We have also developed an Android prototype that implements the download scheduler together with a novel downloader that speeds up the download by exploiting the Fast Start technique. The prototype exhibits the desired properties of the scheduler, and its faster downloading mechanism yields further energy savings of up to 80% compared to the default Android YouTube app.

We present the detailed system model for adaptive modulation and coding (AMC) in the physical layer, and hybrid automatic repeat request (HARQ) in the data link layer, which play key roles in link layer of future wireless technologies as for e.g. LTE-Advanced. The packet combining schemes hard combining and weighted hard combining are compared in an attempt to reduce packet error rate (PER) and increase spectral efficiency. Simulation results show that both combining schemes will always keep the required quality of service (QoS) and increase spectral efficiency significantly over the system without AMC/HARQ. Moreover, with severe interference, hard combining will improve the system spectral efficiency more than weighted hard combining.

Shaping constant bit rate traffic into bursts has been proposed earlier for UDP-based multimedia streaming to save Wi-Fi communication energy of mobile devices. The relationship between the burst size and energy consumption of wireless interfaces is such that the larger is the burst size, the lower is the energy consumption per bit received as long as there is no packet loss. However, the relationship between the burst size and energy in case of TCP traffic has not yet been fully uncovered. In this paper, we develop a power consumption model which describes this relationship in wireless multimedia streaming scenarios. Then, we implement a cross-layer stream delivery system, EStreamer. This system relies on a heuristic derived from the model and on client playback buffer status to determine a burst size and provides as small energy consumption as possible without jeopardizing smooth playback. The heuristic greatly simplifies the deployment of EStreamer compared to most existing solutions by ensuring energy savings regardless of the wireless interface being used. We show that in the best cases using EStreamer reduces energy consumption of a mobile device by 65%, 50-60% and 35% while streaming over Wi-Fi, LTE and 3G respectively. Compared with existing energy-aware applications energy consumption can be reduced by 10-55% further.

Multimedia streaming applications are among the most energy hungry applications in smartphones. The energy consumption mostly depends on the delivery techniques and on the power management techniques of wireless interfaces (Wi-Fi and 3G). In order to provide insights on what kind of streaming techniques exist, how they work on different mobile platforms, and what is their impact on the energy consumption of mobile phones, we have done a large set of active measurements with several smartphones having both Wi-Fi and cellular network access. Our analysis reveals five different techniques to deliver the content to the video players. The selection of a technique depends on the device, player, quality, and service. The results from our power measurements allow us to conclude that none of the identified techniques is optimal because they take none of the following facts into account: access technology used, user behaviour, and user preferences concerning data waste. However, we point out the techniques that provide the most attractive trade-offs in particular situations. Furthermore, we make several observations on the energy consumption of different players, containers, and video qualities that should be taken into consideration when optimizing the energy consumption.

Public cloud platforms might start with homogeneous hardware; nevertheless, because of inevitable hardware upgrades, or adding more capacity, the initial homogeneous platform will gradually evolve into heterogeneous as time passes by. The consequent performance heterogeneity is of concern to cloud users. In this paper, we evaluate performance variations from hardware heterogeneity and scheduling mechanisms of public clouds. Amazon Elastic Compute Cloud (Amazon EC2) and Rackspace Cloud are used as the representatives because of their relatively long record and wide usage among small and medium enterprises (SMEs). A comprehensive set of microbenchmarks and application-level macrobenchmarks have been used to investigate performance variation. Several major contributions have been made. First, we find out that heterogeneous hardware is a commonality among the relatively long-lasting cloud platforms, although the level of heterogeneity varies. Second, we observe that heterogeneous hardware is the primary culprit of performance variation of cloud platforms. Third, we discover that varied CPU acquisition percentages and different virtual machine scheduling mechanisms exacerbate the performance variation problem, especially for network related operations. Finally, based on the observations, we propose cost-saving approaches and analyze Nash equilibrium from cloud user perspective. By using a simple "trial-and-better" approach, i.e., keep good-performing instances and discard bad-performing instances, cloud users can achieve up to 30 percent cost saving.

General-purpose computing domain has experienced strategy transfer from scale-up to scale-out in the past decade. In this paper, we take a step further to analyze ARM-processor based cluster against Intel X86 workstation, from both energy-efficiency and cost-efficiency perspectives. Three applications are selected and evaluated to represent diversified applications, including Web server throughput, in-memory database, and video transcoding. Through detailed measurements, we make the observations that the energy-efficiency ratio of the ARM cluster against the Intel workstation varies from 2.6-9.5 in in-memory database, to approximately 1.3 in Web server application, and 1.21 in video transcoding. We also find out that for the Intel processor that adopts dynamic voltage and frequency scaling (DVFS) techniques, the power consumption is not linear with the CPU utilization level. The maximum energy saving achievable from DVFS is 20%. Finally, by utilizing a monthly cost model of data centers, we conclude that ARM cluster based data centers are feasible, and are advantageous in computationally lightweight applications, e.g. in-memory database and network-bounded Web applications. The cost advantage of ARM cluster diminishes progressively for computation-intensive applications, i.e. dynamic Web server application and video transcoding, because the number of ARM processors needed to provide comparable performance increases.

Detecting and tracking the position of a mobile user is an increasingly important feature in many mobile applications. In this work we study how cheap and energy-efficient air pressure sensors measuring the altitude could be used, as a complement to the dominant GPS system. The cornerstone of our approach is that a huge amount of route data, collected with GPS devices, is available in various cloud services. The location detection and route tracking task thus becomes a question of matching the collected altitude traces with the altitude curves of stored data to find the best matching routes. Here we build a prototype system of crowd-sourced database containing only altitude data. How accurately this stored altitude data could be matched with the collected altitude traces is the key question of our study.

Offloading computation to cloud has been widely used for extending battery life of mobile devices. However, little effort has been invested in applying the offloading techniques to communication-related tasks. We propose SmartDiet, a toolkit to identify the constraints that reduce offloading opportunities and to calculate the energy-saving potential of offloading communication-related tasks. SmartDiet traces the method-level application execution and estimates the allocation of communication energy cost from traffic traces. We discuss key features of SmartDiet and show some preliminary results using a prototype implementation.

This paper studies the electricity consumption of mobile phone chargers. The charger's electricity consumption varies depending on its state. We measured the electricity consumption of various phone and charger models in different states. We also did panel studies on the recharging behavior of smartphone users. Based on these and other sources, we are able to estimate mobile phone recharging electricity consumption, cost, and CO2emissions both in Europe and in the USA. Our analysis shows that the actual recharging of batteries consumes only 40% of the total energy; the rest is wasted mainly by unnecessarily plugged-in chargers consuming 55% of the total energy.

Peer-to-peer based content sharing is considered as one of the most efficient and popular content distribution solution nowadays. However in such network it is common that parts of the content is not distributed to several peers yet, thus during the download we have to wait a lot for that rare pieces. In addition to that it is also possible that we will not be able to download the rare pieces because the original seeder has already left the network. In this paper we investigate how to increase download efficiency in BitTorrent, in case of rare pieces by using network coding methods. In mobile networks the peer connections are not so reliable, because the phones as peers can leave the network any time e.g. if the network coverage is weak. Bringing BitTorrent technology to mobile phones has already been demonstrated with the implementation of SymTorrent and MobTorrent. In this research we investigate how to extend these implementations with network coding, to increase efficiency in case of rare pieces.

We study how much energy can be saved by reshaping audio streaming traffic before receiving at the mobile devices. The rationale is the following: Mobile network interfaces (WLAN and 3G) are in active mode when they transmit or receive data, otherwise they are in idle/sleep mode. To save energy, minimum possible time should be spent in active mode and maximum in idle/sleep mode. It is well known that by reshaping the usually constant bit rate multimedia traffic into bursts, it is possible to spend more time in idle/sleep mode leading to impressive energy savings. We propose a proxy-based solution that shapes an audio stream into bursts before relaying the traffic to the mobile device. The novelty of our work is an evaluation of the energy savings using such a proxy with different configurations for both WLAN access with standard 802.11 Power Saving Mode and 3G access. We conclude that for WLAN access, proxy causes power savings of 30%-65% depending on the audio stream rate, location of the proxy and amount of cross traffic. In the case of 3G, the effectiveness of our proxy seems to vary depending on the phone model and operator. In some cases, the energy savings are encouraging, while in other cases the proxy turns out to be ineffective due to abnormal delay variation and TCP flow control behavior.

Using proxy servers to cache and shape network traffic can significantly improve the energy efficiency of the participating mobile clients. In this paper, we analyze the implications of hosting a BitTorrent proxy on a broadband router, which pushes the content to a mobile phone over wireless radio (WLAN or 3G). The amount of memory in a router is limited and therefore our interest is on how to use efficiently the memory to download the BitTorrent content as fast as possible and at the same time transfer it to the mobile phone in an energy-efficient way. We investigate these aspects via a series of measurements. The results show that the proxy-based solution outperforms the torrent client running on the phone in terms of energy and download time. We also simulate the BitTorrent operation to understand how these memory-limited devices would influence the operation of the whole community.

BitTorrent-based file-sharing is already available for mobile phones; however, its energy profile makes it difficult to use it for transferring large amount of data. This paper analyzes an alternative cloud-based solution that uses a remote server to download content via BitTorrent and transfer it to the mobile device in a transparent and energy efficient way. The system is evaluated via measurements carried out on smartphones.

In this paper we investigate the DHT performance for Peer-to-peer SIP. We analyze performance from the mobile phone perspective. This angle brings battery consumption and efficient use of the limited resources of the mobile devices to the focus. We develop an analytical model of Kademlia DHT performance and use it to analyze a number of different strategies to use peer-to-peer SIP with mobile devices. These analyses include studying a model where all devices participate in the DHT and contrasting that with a model where only a subset of devices forms the DHT. Our analysis shows that in a one million user P2PSIP network, at least 30% of the mobile nodes should participate in operating the DHT to remain energy-efficient.

In this paper, we analyze the end-to-end communication activities of a modern mobile phone, Nokia N95, to understand how much energy different communication alternatives consume. In particular, we investigate the interactions when multiple connections are used in parallel. Parallel connections save energy but the gains vary depending on the technology. TCP downloads during 3G voice calls result into 75%-90% energy savings, TCP downloads during VoIP calls result into 30%-40% savings, and TCP downloads when other TCP streams are active at the same interface result into 0%20% savings. The results indicate that there is a significant potential to save energy if applications are engineered to take advantage of this phenomenon.

We investigated consumer attitudes towards energy consumption of mobile phones and mobile services with a questionnaire study among Finnish university students (N = 150). Our questions covered topics like how battery consumption affected phone and application selection and configuration, what effect different battery levels had on user behavior, what kind of recharging policies were used, how satisfied users were with energy consumption feedback, and what kind of wishes for improvements they had. Battery consumption is clearly important for users when buying mobile phones and choosing applications to use. Importance of energy awareness, energy-driven customization, and energy as a decision criterion stood out as other key themes among the replies.

To understand the technical feasibility and user perception of mobile video, we conducted a two-week trial where a student community used mobile phone and Web technologies to share the events of a campus festival. The questions we investigated included the capability of contemporary smart phones to capture and share the events via real time and stored video; the usage patterns that arise within mobile video community; and the social connections between shooters and viewers. Outcomes on the technical level indicate that challenges remain. The bandwidth of 3G cellular connections limits the quality of live videos and video transfer via public WLAN networks is unreliable causing lost connections. The best results today can be achieved by sending the captured videos from mobile phones to the server in the background and providing video-on-demand rather than live video service. The observed consumption patterns show that freshness of the videos is the least important selection factor while manual and automatic textual tagging is important. Seamless integration of video sharing to camera application would improve the user experience.

BitTorrent is an efficient peer-to-peer content sharing protocol. In this paper we investigate how this popular technology can be brought to mobile handheld devices. As handheld devices are increasingly used for music and video consumption it is reasonable to consider how a handheld device can join existing communities and be used to find, access, and play multimedia content. The paper is based on the experiences of BitTorrent implementation on mainstream mobile phones (Nokia Series 40) over Java Micro Edition platform. The paper analyses the important implementation issues of such large peer-to-peer application on low end mobile devices and shares practical experiences. The performance of the application is evaluated by speed and memory usage measurements.

Battery consumption is an important but not very widely studied aspect of peer-to-peer applications. This paper describes a series of experiments which obtained detailed measurements of BitTorrent energy consumption on handheld devices. The measurements that were carried out with SymTorrent client on three different models of Nokia S60 mobile phones indicate that peer-to-peer content sharing on handhelds is practical from the energy consumption point of view. The results also show that acting as a true peer and uploading content for other users does not cause much extra energy consumption during active downloading.

We will demonstrate how an existing content management system, Drupal, can be used on a personal Web site that is running on a mobile phone. It allows users to easily create and configure a Web site to their mobile devices that is able to take advantage of the wide set of normal functionality of the content management system and the context and data of the mobile phone. We estimate that very easy creation of versatile mobile Web sites has potential to boost the development of innovative mobile applications.

During the last years progress in Web search engines has been made to the point that relevant information can be reached easily most of the time. However very little empirical research has been carried to study Web search in highly dynamic social network environments composed of mobile devices. The aim of this work was therefore to investigate novel approaches that took advantage of the social network environment inherent to mobile peer-to-peer paradigm. The work focused mainly on the development of a prototype for mobile search concept. The prototype was built on top of Drupal content site management system. This study suggests that the methods presented can be a complement to traditional Web search engines.

We will demonstrate how a common mainstream web application development stack, the so-called AMP stack, can be installed on a mobile phone and used for creating a mobile website that utilize the unique characteristics of a mobile personal device. By providing a widely used web application development environment also on the mobile phone, we foresee that the threshold for developing mobile Web sites effectively can be removed.

Peer-to-peer applications are emerging into mobile devices. However, resource limitations of these devices introduce new challenges for P2P technologies. For instance, there is a need for incentive mechanisms, which address the free riding problem but do not waste devices' battery or communication resources. A centralized and user-identity based incentive mechanism enables mobile users to contribute with any device and receive P2P services with mobile devices. We explore security issues related to a centralized incentive mechanism by analyzing and classifying threats and potential security mechanisms. We propose a privacy preserving security architecture. The architecture is based on authentication, software tamper protection, and misbehavior detection mechanisms. Further, we provide a discussion on potential security compromises, not jeopardizing sufficient security level, and describe a prototype implementation for mobile BitTorrent file sharing peers.

In this paper we describe four P2P applications developed for smart phones, ranging from distributed computing to content sharing and search in closed and open user groups. Their common feature relies in the communication interface which has been cellular data transmission (SMS, HSCSD, respectively GPRS). Through our examples we highlight various aspects that application development on smart phones has to take into account, such as traffic efficiency, CPU load and social awareness. We discuss our main observations related to the experiments and highlight some promising directions for future mobile P2P applications. The discussed set of applications developed by us in the period between 2001-2005 cover the areas of distributed computing, large-scale content sharing in open communities, content sharing in closed groups, respectively keyword search in a social network of phones

A very efficient peer-to-peer application layer architecture is analyzed as a potential candidate for wireless peer-to-peer applications. Its performance in terms of generated traffic and load balance are simulated for different network sizes. A number of candidate cluster topologies are proposed. Based on the simulation results, the optimal cluster topology and cluster size are identified. We conclude that cluster sizes that are the square root of the number of nodes generate uniform traffic. The cluster topology should be star, ring or a compromise between the two, 'planned N'. We show that, for these topologies, the traffic increases less than linearly with the number of nodes in the network, making it highly scalable. Our conclusions are valid for uniform query and update distributions. In addition to the approach from earlier work (Csucs, G. et al., 2002), maintenance aspects are also dealt with; we briefly describe the basic link management procedures that make such networks feasible, but we do not cover their performance analysis. Neither do we treat query topologies.

The coupling of symbolic and numeric computing in design is discussed. Symbolic computing is used to facilitate the use of complicated numerical design tools such as simulators. Rather than studying new analysis techniques, the author developed a framework which makes possible the easy and efficient use of existing tools. Application and tool-specific knowledge together with a graphical user-interface take care of design routines such as conversions and error detection and correction. This helps the user to operate the tools in an easy way and, additionally, facilitates the implementation of higher-level operations for design synthesis and optimization. The techniques have been applied in a design system for mobile telephones and radio links which is currently in field-testing.>