Efficient container handling and early damage detection are critical for minimizing operational delays, reducing costs, and ensuring safety in global maritime logistics.This work presents a deep learning-based methodology for real-time container tracking and automated damage detection during crane unloading operations at container terminals.We develop and deploy two specialized YOLOv12-based object detection models: one for identifying containers in motion and another for detecting structural damages such as bents, dents, and holes.Our models are trained and evaluated on a real-world dataset curated from video feeds captured at the EUROGATE Container Terminal in Limassol, Cyprus.The system is designed for robust performance under realistic terminal conditions, including variable lighting and motion.Our models achieve high detection accuracy, with a mAP50 of 0.99 for container detection and 0.75 for damage detection, substantially outperforming existing benchmarks.These results highlight the practical potential of our method for improving efficiency and safety in automated maritime logistics.

This study investigates the combined berth allocation problem (BAP) and quay crane allocation problem (QCAP) while considering a multi-quay setting. First, a mixed integer linear programming mathematical model is developed based on various constraints and real port settings. Then, the multi-quay combined BAP and QCAP is solved using both the exact method and a metaheuristic optimization method, namely, the cuckoo search algorithm (CSA). This analysis pertains to a one-week planning scenario, utilizing data from a real port. The results of the comparative analysis show that the proposed CSA can provide a near-optimal solution (< 1.02% from the optimal) at a fraction of the computational time (10 times faster), as compared to the exact solution. This makes it suitable for solving larger instances of the combined BAP and QCAP for bigger terminals and extended planning horizons.

Modern data storage systems are characterized by a distributed architecture as well as the presence of multiple storage tiers and caches. Both system developers and operators are challenged with the complexity of such systems as it is hard to evaluate how a configuration change will impact the workload or system performance and identify the best configuration to satisfy some performance objective. DITIS is a new simulator that models the end-to-end execution of file requests on distributed tiered storage systems that addresses the aforementioned challenges efficiently without any costly system redeployments. The demonstration will showcase the key functionalities and benefits offered by DITIS, including (i) analyzing workload traces to understand their characteristics and the behavior of the underlying storage system; (ii) running simulations with different configurations to evaluate their impact on performance; and (iii) running optimizations over custom search spaces to find the best configuration that satisfies a given objective.

The Internet of things (IoT) ecosystem provides a platform for the connectivity of interrelated smart devices to automate manual processes and reduce labor costs.IoT has brought significant benefits to all industries, including maritime, as various objects (e.g., ports, ships, agents, etc.) are connected to gather and share information within the maritime ecosystem.The innovative technological aspects of IoT are promoting the effective collaboration between the research community and the maritime industry, for enhancing the performance of maritime transportation systems.Therefore, this study discusses recent advances delivered by the IoT and other emerging technologies, like machine learning (ML) and computer vision (CV), for smart maritime transportation systems (SMTSs).In particular, this paper presents two specific use cases of SMTSs, namely, predictive maintenance and container damage/seal inspection.Moreover, the key benefits of integrating IoT with ML and CV are highlighted for the above-mentioned use cases.Finally, a discussion is presented to highlight key opportunities along with foreseeable future challenges in adopting these new technologies by the maritime industry.

The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN’s resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.

Distributed data stream processing systems (DSPSs) such as Storm, Flink, and Spark Streaming are now routinely used to process continuous data streams in (near) real-time. However, achieving the low latency and high throughput demanded by today's streaming applications can be a daunting task, especially since the performance of DSPSs highly depends on a large number of system parameters that control load balancing, degree of parallelism, buffer sizes, and various other aspects of system execution. This tutorial offers a comprehensive review of the state-of-the-art automatic performance tuning approaches that have been proposed in recent years. The approaches are organized into five main categories based on their methodologies and features: cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. The categories of approaches will be analyzed in depth and compared to each other, exposing their various strengths and weaknesses. Finally, we will identify several open research problems and challenges related to automatic performance tuning for DSPSs.

The Automatic Identification System (AIS) provides information for tracking and monitoring vessel activity in real time. The vessel traffic data from AIS includes position coordinates in latitude and longitude, speed and course over ground, the vessel's unique identification number, and many more. In this work, we investigate the use of AIS data for estimating sea surface current velocities in the Eastern Mediterranean sea. Specifically, we apply the dead reckoning technique to compute the difference between the projected position and the true position of a vessel over time, which is mainly attributed on the force of sea surface currents. The estimated sea surface current velocities and directions are compared with the ones provided by the Copernicus Marine ocean product system. The analysis reveals that the dead reckoning technique can be used reliably for estimating sea currents at a very fine granularity, especially in high-traffic and coastal areas, where there is an increased complexity of obtaining accurate results from other sources.

Automatic identification system (AIS) data provides a wealth of information regarding vessel traffic and is used for a variety of applications such as collision detection and avoidance, route prediction and optimization, search and rescue operations, etc. However, several challenges exist when working with AIS data including huge volume and velocity (as AIS signals are sent by vessels every few seconds), message duplication, various types of data irregularities, as well as the need for real-time processing and analysis. This paper presents a new framework for collecting, processing, storing, and analyzing AIS data in real time plus a set of algorithms for doing so in an efficient and scalable way. At the same time, a set of intelligent services are provided as building blocks for improving and creating new AIS data driven applications. This framework has been operational for the past few years in Cyprus, and has collected and processed around one billion AIS messages from the Eastern Mediterranean Sea.

The port call process encapsulates a visitation cycle of a ship to a port and can generate a wealth of data. The real time analysis of port call data can be used to find bottlenecks in the port call process, establish targets based on key performance indicators (KPIs), and to understand how shipping traffic impacts a port's efficiency. This demonstration will showcase a new Power BI interactive report powered by a multidimensional OLAP cube for very fast performance, which is built on top of a data warehouse collecting information from various sources in real time. The report currently visualizes several KPIs and other types of information that can be filtered per port, time-period, vessel type, origin or destination ports, and various other categories to help manage arrivals, departures, and port operations.

Proper utilization of port resources and efficient berth planning play a crucial role in minimizing port congestion and overall handling costs. Therefore, this study focuses on efficient berth planning in maritime container terminals composed of multiple quays. In particular, this study addresses the Multi-Quay Berth Allocation Problem (MQ-BAP), where a continuous berthing layout is considered along with dynamic ship arrivals and practical constraints such as safety time windows and safety distances between ships. Since MQ-BAP is an NP-hard problem, this study proposes a metaheuristic-based approach, the Cuckoo Search Algorithm (CSA) for solving the problem. A comparative study is also performed using real data instances collected from the Port of Limassol, Cyprus, against a genetic algorithm solution proposed in the recent literature, as well as the optimal exact solution implemented using MILP. The results of the experiments show the effectiveness of our proposed CSA approach in handling real-world berth allocation in ports with multiple quays while also considering practical constraints.

With the advent of the Internet and Internet-connected devices, modern business applications can experience rapid increases as well as variability in transactional workloads. Database replication has been employed to scale performance and improve availability of relational databases but past approaches have suffered from various issues including limited scalability, performance versus consistency tradeoffs, and requirements for database or application modifications. This paper presents Hihooi, a replication-based middleware system that is able to achieve workload scalability, strong consistency guarantees, and elasticity for existing transactional databases at a low cost. A novel replication algorithm enables Hihooi to propagate database modifications asynchronously to all replicas at high speeds, while ensuring that all replicas are consistent. At the same time, a fine-grained routing algorithm is used to load balance incoming transactions to available replicas in a consistent way. Our thorough experimental evaluation with several well-established benchmarks shows how Hihooi is able to achieve almost linear workload scalability for transactional databases.

Aggression on social media has evolved into a major point of concern. However, recently proposed machine learning (ML) approaches to detect various types of aggressive behavior fall short, due to the fast and increasing pace of content generation as well as evolution of such behavior over time. This work introduces the first, practical, real-time framework for detecting aggression on Twitter via embracing the streaming ML paradigm. This method adapts its ML binary classifiers in an incremental fashion, while receiving new annotated examples, and achieves similar performance as batch-based ML models, with 82-93% accuracy, precision, and recall. Experimental analysis on real Twitter data reveals how this framework, implemented in Spark Streaming, easily scales to process millions of tweets in minutes.

Database replication can be employed for scaling transactional workloads while maintaining strong consistency semantics. However, past approaches suffer from various issues such as limited scalability, performance versus consistency tradeoffs, and requirements for database or application modifications. Hihooi is a new replication-based master-slave middleware system that is able to overcome the aforementioned limitations. The novelty of Hihooi lies in its modern architecture as well as its replication and transaction routing algorithms. In particular, Hihooi replicates all write statements asynchronously and applies them in parallel at the replica nodes, while ensuring replica consistency. At the same time, a fine-grained transaction routing algorithm ensures that all read transactions are load balanced to the replicas consistently. This demonstration will showcase the key functionalities of Hihooi, including (i) practical management of system components and databases (e.g., add a new replica node), (ii) increased scalability compared to state-of-the-art approaches, and (iii) support for elasticity by suspending and resuming database replicas online without service interruption.

Tourists’ revisit has significant monetary benefits to destinations because the cost of retaining existing visitors is less than attracting new visitors. Re-visit intention is often based on tourists experience and satisfaction at a destination. An important aspect that influences the relationship between satisfaction and intention to revisit is the weather conditions at a destination given the increased frequency of heatwaves that strike summer holiday destinations over the summer months. This work applies natural language processing and classification techniques to evaluate the impact of weather information on revisit intention utilizing reviews from TripAdvisor and online weather data. Information retrieval techniques (Doc2Vec) are applied on online reviews collected during the summer months between 2010-2019 from tourists that visited Cyprus. Reviews are labeled as "revisits" or "neutral" based on their textual content. The labelled reviews dataset is enhanced with weather information based on the reviews’ timestamp, such as temperature and humidity of tourists’ country of origin and Cyprus at the time of the visit to the hotel/destination. To account for the influence of hotel infrastructure and available services to deal with heatwaves (i.e., climate-controlled), the training dataset included hotel star rating as an additional parameter. An ensemble gradient boosting tree classifier is trained utilizing the compiled dataset to predict revisit intention. The classifier is evaluated against the area under the curve. To interpret the classifier’s inherent patterns, a popular machine learning interpretation technique is used, namely Shapley Additive Explanation (SHAP). Visualizations of the model using SHAP indicate that the heat index and weather difference between destination and country of origin influence revisit intention. Such preliminary insights are encouraging for further investigations with an end goal to develop a decision support system to assist destination managers during their target marketing campaigns.

Over the last couple of decades, demand for seaborne containerized trade has increased significantly and it is expected to continue growing over the coming years. As an important node in the maritime industry, a maritime container terminal (MCT) should be able to tackle the growing demand for sea trade. Due to the increased number of ships that can arrive simultaneously at an MCT combined with inefficient berth allocation procedures, there are often undesirable situations when the ships have to stay in waiting queues and delay both their berthing and departure. In order to improve port efficiency in terms of reducing the total handling cost and late departures, this study investigates the dynamic and continuous berth allocation problem (DC-BAP), where vessels are assigned dynamically as they arrive at their berth locations assuming a continuous berth layout. First, the DC-BAP is formulated as a mixed-integer linear programming (MILP) model. Since BAP is an NP-hard problem and cannot be solved by mathematical approaches in a reasonable time, this study adopts the recently developed metaheuristic cuckoo search algorithm (CSA) to solve the DC-BAP. For validating the performance of the proposed CSA method, we use a benchmark case study and a genetic algorithm solution proposed in recent literature as well as compare our results against the optimal MILP solution. From the simulation results, it becomes evident that the newly proposed algorithm has higher efficiency over counterparts in terms of optimal berth allocation within reasonable computation time.

The recent advancements in storage technologies have popularized the use of tiered storage systems in data-intensive compute clusters. The Hadoop Distributed File System (HDFS), for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, the task schedulers of big data platforms (such as Hadoop and Spark) will assign tasks to available resources only based on data locality information, and completely ignore the fact that local data is now stored on a variety of storage media with different performance characteristics. This paper presents Trident, a principled task scheduling approach that is designed to make optimal task assignment decisions based on both locality and storage tier information. Trident formulates task scheduling as a minimum cost maximum matching problem in a bipartite graph and uses a standard solver for finding the optimal solution. In addition, Trident utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. Trident is implemented in both Spark and Hadoop, and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.

The rise of online aggression on social media is evolving into a major point of concern. Several machine and deep learning approaches have been proposed recently for detecting various types of aggressive behavior. However, social media are fast paced, generating an increasing amount of content, while aggressive behavior evolves over time. In this work, we introduce the first, practical, real-time framework for detecting aggression on Twitter via embracing the streaming machine learning paradigm. Our method adapts its ML classifiers in an incremental fashion as it receives new annotated examples and is able to achieve the same (or even higher) performance as batch-based ML models, with over 90% accuracy, precision, and recall. At the same time, our experimental analysis on real Twitter data reveals how our framework can easily scale to accommodate the entire Twitter Firehose (of 778 million tweets per day) with only 3 commodity machines. Finally, we show that our framework is general enough to detect other related behaviors such as sarcasm, racism, and sexism in real time.

The Internet of Things (IoT) extends connectivity beyond traditional computing devices to different types of smart objects, equipped with various sensors and actuators. These objects range from smart lightbulbs and thermostats to smart watches and fitness trackers, or even heavy machinery used in various industrial sectors. Due to device heterogeneity, the complexity of developing applications that require the collection and sharing of data across multiple IoT devices is high, as developers need to be familiar with a diverse set of supported services and APIs. While existing approaches have proposed solutions to this challenge, they rely on the use of resource-intensive cloud-based components, they do not offer the degree of extensibility desired by developers, and they often trade off some of the richness of real-time data for ease of use. Cuttlefish is a flexible and lightweight middleware that offers a unified API to help with the development of applications that utilize multiple heterogeneous IoT devices. It abstracts away much of the complexity involved with orchestrating different devices at runtime. At the same time, it avoids the aforementioned caveats of existing approaches through a simple and efficient design, yet one that offers a rich set of capabilities to developers.

The recent emergence of Internet-of-Things (IoT) technologies in mission-critical applications in the maritime industry has led to the introduction of the Internet-of-Ships (IoS) paradigm. IoS is a novel application domain of IoT that refers to the network of smart interconnected maritime objects, which can be any physical device or infrastructure associated with a ship, a port, or the transportation itself, with the goal of significantly boosting the shipping industry toward improved safety, efficiency, and environmental sustainability. In this article, we provide a comprehensive survey of the IoS paradigm, its architecture, its key elements, and its main characteristics. Furthermore, we review the state of the art for its emerging applications, including safety enhancements, route planning and optimization, collaborative decision making, automatic fault detection and preemptive maintenance, cargo tracking, environmental monitoring, energy-efficient operations, and automatic berthing. Finally, the presented open challenges and future opportunities for research in the areas of satellite communications, security, privacy, maritime data collection, data management, and analytics, provide a road map toward optimized maritime operations and autonomous shipping.

Data-intensive platforms such as Hadoop and Spark are routinely used to process massive amounts of data residing on distributed file systems like HDFS. Increasing memory sizes and new hardware technologies (e.g., NVRAM, SSDs) have recently led to the introduction of storage tiering in such settings. However, users are now burdened with the additional complexity of managing the multiple storage tiers and the data residing on them while trying to optimize their workloads. In this paper, we develop a general framework for automatically moving data across the available storage tiers in distributed file systems. Moreover, we employ machine learning for tracking and predicting file access patterns, which we use to decide when and which data to move up or down the storage tiers for increasing system performance. Our approach uses incremental learning to dynamically refine the models with new file accesses, allowing them to naturally adjust and adapt to workload changes over time. Our extensive evaluation using realistic workloads derived from Facebook and CMU traces compares our approach with several other policies and showcases significant benefits in terms of both workload performance and cluster efficiency.

Database and big data analytics systems such as Hadoop and Spark have a large number of configuration parameters that control memory distribution, I/O optimization, parallelism, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators struggle to understand and tune them to achieve good performance. In this tutorial, we review existing approaches on automatic parameter tuning for databases, Hadoop, and Spark, which we classify into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We describe the foundations of different automatic parameter tuning algorithms and present pros and cons of each approach. We also highlight real-world applications and systems, and identify research challenges for handling cloud services, resource heterogeneity, and real-time analytics.

The uptake of wearable technology suggests that the time is ripe to explore new opportunities for improving mobile experiences. Apps, however, are not keeping up with the pace of technological advancement because wearables are treated as standalone devices, although their individual capabilities better classify them as peripherals with complementary roles. We foresee that the next generation of apps will orchestrate multiple wearable devices to enhance mobile user experiences. However, currently there is limited support for combining heterogeneous devices. This paper introduces Multi-Wear, a platform to scaffold the development of apps that span multiple wearables. It demonstrates experimentally how MULTI-WEAR can help bring changes to mobile apps that go beyond conventional practices.

The continuous improvements in memory, storage devices, and network technologies of commodity hardware introduce new challenges and opportunities in tiered storage management. Whereas past work is exploiting storage tiers in pairs or for specific applications, OctopusFS---a novel distributed file system that is aware of the underlying storage media---offers a comprehensive solution to managing multiple storage tiers in a distributed setting. OctopusFS contains auto-mated data-driven policies for managing the placement and retrieval of data across the nodes and storage tiers of the cluster. It also exposes the network locations and storage tiers of the data in order to allow higher-level systems to make locality-aware and tier-aware decisions. This demonstration will showcase the web interface of OctopusFS, which enables users to (i) view detailed utilization information for the various storage tiers and nodes, (ii) browse the directory namespace and perform file-related actions, and (iii) execute caching-related operations while observing their performance impact on MapReduce and Spark workloads.

The ever-growing data storage and I/O demands of modern large-scale data analytics are challenging the current distributed storage systems. A promising trend is to exploit the recent improvements in memory, storage media, and networks for sustaining high performance and low cost. While past work explores using memory or SSDs as local storage or combine local with network-attached storage in cluster computing, this work focuses on managing multiple storage tiers in a distributed setting. We present OctopusFS, a novel distributed file system that is aware of heterogeneous storage media (e.g., memory, SSDs, HDDs, NAS) with different capacities and performance characteristics. The system offers a variety of pluggable policies for automating data management across the storage tiers and cluster nodes. The policies employ multi-objective optimization techniques for making intelligent data management decisions based on the requirements of fault tolerance, data and load balancing, and throughput maximization. At the same time, the storage media are explicitly exposed to users and applications, allowing them to choose the distribution and placement of replicas in the cluster based on their own performance and fault tolerance requirements. Our extensive evaluation shows the immediate benefits of using OctopusFS with data-intensive processing systems, such as Hadoop and Spark, in terms of both increased performance and better cluster utilization.

The proliferation of wearable and smartphone devices with embedded sensors has enabled researchers and engineers to study and understand user behavior at an extremely high fidelity, particularly for use in industries such as entertainment, health, and retail. However, identified user patterns are yet to be integrated into modern systems with immersive capabilities, such as VR systems, which still remain constrained by limited application interaction models exposed to developers. In this paper, we present Smart VR, a platform that allows developers to seamlessly incorporate user behavior into VR apps. We present the high-level architecture of Smart VR, and show how it facilitates communication, data acquisition, and context recognition between smart wearable devices and mediator systems (e.g., smartphones, tablets, PCs). We demonstrate Smart VR in the context of a VR app for retail stores to show how it can be used to substitute the requirement of cumbersome input devices (e.g., mouse, keyboard) with more natural means of user-app interaction (e.g., user gestures such as swiping and tapping) to improve user experience.

The MapReduce programming model has become widely adopted for large scale analytics on big data. MapReduce systems such as Hadoop have many tuning parameters, many of which have a significant impact on performance. The map and reduce functions that make up a MapReduce job are developed using arbitrary programming constructs, which makes them black-box in nature and prevents users from making good parameter tuning decisions for a submitted MapReduce job. Some research projects, such as the Starfish system, aim to provide automatic tuning decisions for input MapReduce jobs. Starfish and similar systems rely on an execution profile of a MapReduce job being tuned, and this profile is assumed to come from a previous execution of the same job. Managing these execution profiles has not been previously studied. This thesis presents PStorM, a profile store that organizes the collected profiling information in a scalable and extensible data model, and a profile matcher that accurately picks the relevant profiling information even for previously unseen MapReduce jobs. PStorM is currently integrated with the Starfish system, providing the necessary profiles that Starfish needs to tune a job. The thesis presents results that demonstrate the accuracy and efficiency of profile matching. The results also show that the profiles returned by PStorM lead to Starfish tuning decisions that are as good as the decisions made by profiles collected from a previous run of the job.

Large-scale data center networks are complex---comprising several thousand network devices and several hundred thousand links---and form the critical infrastructure upon which all higher-level services depend on. Despite the built-in redundancy in data center networks, performance issues and device or link failures in the network can lead to user-perceived service interruptions. Therefore, determining and localizing user-impacting availability and performance issues in the network in near real time is crucial. Traditionally, both passive and active monitoring approaches have been used for failure localization. However, data from passive monitoring is often too noisy and does not effectively capture silent or gray failures, whereas active monitoring is potent in detecting faults but limited in its ability to isolate the exact fault location depending on its scale and granularity.

There is a growing trend of performing analysis on large datasets using workflows composed of MapReduce jobs connected through producer-consumer relationships based on data. This trend has spurred the development of a number of interfaces---ranging from program-based to query-based interfaces---for generating MapReduce workflows. Studies have shown that the gap in performance can be quite large between optimized and unoptimized workflows. However, automatic cost-based optimization of MapReduce workflows remains a challenge due to the multitude of interfaces, large size of the execution plan space, and the frequent unavailability of all types of information needed for optimization. We introduce a comprehensive plan space for MapReduce workflows generated by popular workflow generators. We then propose Stubby , a cost-based optimizer that searches selectively through the subspace of the full plan space that can be enumerated correctly and costed based on the information available in any given setting. Stubby enumerates the plan space based on plan-to-plan transformations and an efficient search algorithm. Stubby is designed to be extensible to new interfaces and new types of optimizations, which is a desirable feature given how rapidly MapReduce systems are evolving. Stubby's efficiency and effectiveness have been evaluated using representative workflows from many domains.

Infrastructure-as-a-Service (IaaS) cloud platforms have brought two unprecedented changes to cluster provisioning practices. First, any (nonexpert) user can provision a cluster of any size on the cloud within minutes to run her data-processing jobs. The user can terminate the cluster once her jobs complete, and she needs to pay only for the resources used and duration of use. Second, cloud platforms enable users to bypass the traditional middleman---the system administrator---in the cluster-provisioning process. These changes give tremendous power to the user, but place a major burden on her shoulders. The user is now faced regularly with complex cluster sizing problems that involve finding the cluster size, the type of resources to use in the cluster from the large number of choices offered by current IaaS cloud platforms, and the job configurations that best meet the performance needs of her workload.

Table partitioning splits a table into smaller parts that can be accessed, stored, and maintained independent of one another. From their traditional use in improving query performance, partitioning strategies have evolved into a powerful mechanism to improve the overall manageability of database systems. Table partitioning simplifies administrative tasks like data loading, removal, backup, statistics maintenance, and storage provisioning. Query language extensions now enable applications and user queries to specify how their results should be partitioned for further use. However, query optimization techniques have not kept pace with the rapid advances in usage and user control of table partitioning. We address this gap by developing new techniques to generate efficient plans for SQL queries involving multiway joins over partitioned tables. Our techniques are designed for easy incorporation into bottom-up query optimizers that are in wide use today. We have prototyped these techniques in the PostgreSQL optimizer. An extensive evaluation shows that our partition-aware optimization techniques, with low optimization overhead, generate plans that can be an order of magnitude better than plans produced by current optimizers.

The need to improve a suboptimal execution plan picked by the query optimizer for a repeatedly run SQL query arises routinely. Complex expressions, skewed or correlated data, and changing conditions can cause the optimizer to make mistakes. For example, the optimizer may pick a poor join order, overlook an important index, use a nested-loop join when a hash join would have done better, or cause an expensive, but avoidable, sort to happen. SQL tuning is also needed while tuning multi-tier services to meet service-level objectives. The difficulty of SQL tuning can be lessened considerably if users and higher-level tuning tools can tell the optimizer: "I am not satisfied with the performance of the plan p being used for the query Q that runs repeatedly. Can you generate a (δ%) better plan?" This paper designs, implements, and evaluates Xplus which, to our knowledge, is the first query optimizer to provide this feature. Xplus goes beyond the traditional plan-first-execute-next approach: Xplus runs some (sub)plans proactively, collects monitoring data from the runs, and iterates. A nontrivial challenge is in choosing a small set of plans to run. Xplus guides this process efficiently using an extensible architecture comprising SQL-tuning experts with different goals, and a policy to arbitrate among the experts. We show the effectiveness of Xplus on real-life tuning scenarios created using TPC-H queries on a PostgreSQL database.