Many Internet of Things (IoT) applications require compute resources that cannot be provided by the devices themselves. On the other hand, processing of the data generated by IoT devices and sensors often has to be performed in real- or near real-time, i.e., with stringent latency requirements in constrained environments (e.g., intermittent network connectivity and limited power envelopes). Examples of such scenarios are autonomous vehicles in the form of cars and drones where the processing and analysis of observational data (e.g., video feeds) need to be performed expeditiously to allow for safe operation of the vehicles and to deliver the results in a timely fashion to the stakeholders of the mission. To support the compute and timeliness requirements of such applications, it is essential to include suitable edge resources to process these workflows, and to develop an end-to-end system that can route the vehicles dynamically and process and deliver mission-critical data and analyzed results. In this paper, we develop and evaluate a dynamic scheduling approach that considers complex tradeoffs between real-time constraints, network availability, and latency sensitivity of the mission. We devise an optimized route planning and data transmission schedule for drone flights. The scheduling algorithm is encapsulated in a novel end-to-end architecture (FlyPaw) and an associated adaptive drone mission control system, which enables deployment and management of an integrated cyberphysical system (CPS) – from real drone testbed to base stations to edge-to-cloud resources. The planning algorithm takes into account measured network communication characteristics, estimated uncertainties of future data link connectivity, and data timeliness requirements of the mission to prioritize candidate decision tree solutions based on a risk metric derived from Sharpe's ratio. Our results show that for given task sets, Net Time to Retrieve, our metric describing the time required to perform end-to-end collection and downstream processing of data, can be significantly reduced compared to other naive approaches. The theoretical improvement provided by our algorithm over other naive approaches is dependent on several factors — task locations, network connectivity, processing times and available resources, and is bounded by the duration of the drone flight.

In this paper, we describe how we extended the Pegasus Workflow Management System to support edge-to-cloud workflows in an automated fashion. We discuss how Pegasus and HTCondor (its job scheduler) work together to enable this automation. We use HTCondor to form heterogeneous pools of compute resources and Pegasus to plan the workflow onto these resources and manage containers and data movement for executing workflows in hybrid edge-cloud environments. We then show how Pegasus can be used to evaluate the execution of workflows running on edge only, cloud only, and edge-cloud hybrid environments. Using the Chameleon Cloud testbed to set up and configure an edge-cloud environment, we use Pegasus to benchmark the executions of one synthetic workflow and two production workflows: CASA-Wind and the Ocean Observatories Initiative Orcasound workflow, all of which derive their data from edge devices. We present the performance impact on workflow runs of job and data placement strategies employed by Pegasus when configured to run in the above three execution environments. Results show that the synthetic workflow performs best in an edge only environment, while the CASA - Wind and Orcasound workflows see significant improvements in overall makespan when run in a cloud only environment. The results demonstrate that Pegasus can be used to automate edge-to-cloud science workflows and the workflow provenance data collection capabilities of the Pegasus monitoring daemon enable computer scientists to conduct edge-to-cloud research.

Accurate and timely prediction of flash flooding events can be a very useful tool for stormwater officials and first responders. Having lead time with which to issue evacuation directives, to close flood prone roadways, to deploy rescue gear and personnel, and to fortify areas against flooding is essential to minimize property damage and risk of casualties. In this poster, we are presenting a flash flooding prediction workflow based on the Hydrology Lab-Research Distributed Hydrologic Model (HL-RDHM). This workflow leverages cloud computing and the Pegasus Workflow Management System to provide continuous high resolution flood predictions for the Dallas-Fort Worth Metroplex area in North Texas, and can be easily expanded to other regions.

Future drone and air taxi services will take place in the lowest parts of the atmosphere. In the United States, this region is vastly under sampled by existing atmospheric sensing systems and in-situ sensors. As a result, current and future vehicle operators may lack situational awareness of changing weather conditions, bringing uncertainty to aerial ride sharing, cargo delivery and emergency services, impacting up-time, business viability and safety of both vehicles and resources on the ground. This paper presents interviews with drone and air taxi operators on their weather related needs and current sources of weather information as a first step in defining requirements for integrative remote sensing solutions. The solution system is being prototyped and demonstrated in North Texas in collaboration with universities, government agencies and drone operators.

Adverse weather disrupts routine manned aviation operations causing congestion, flight delays and economic losses. However worldwide air travel remains relatively safe and accidents are infrequent due to robust weather observation infrastructure, air traffic management services and regulatory oversight agencies that prioritize a culture of safety. While the introduction and adoption of Unmanned Aircraft Systems is unleashing a new generation of products and services across the world, many unique challenges remain in ensuring safe and efficient flights under unfavorable weather conditions, opening up new areas of research and development. This paper discusses some of these research topics and presents progress being made by the authors in developing end-to-end weather observation and avoidance systems.

Computational science today depends on complex, data-intensive applications operating on datasets from a variety of scientific instruments. A major challenge is the integration of data into the scientist's workflow. Recent advances in dynamic, networked cloud resources provide the building blocks to construct reconfigurable, end-to-end infrastructure that can increase scientific productivity. However, applications have not adequately taken advantage of these advanced capabilities. In this work, we have developed a novel network-centric platform that enables high-performance, adaptive data flows and coordinated access to distributed cloud resources and data repositories for atmospheric scientists. We demonstrate the effectiveness of our approach by evaluating time-critical, adaptive weather sensing workflows, which utilize advanced networked infrastructure to ingest live weather data from radars and compute data products used for timely response to weather events. The workflows are orchestrated by the Pegasus workflow management system and were chosen because of their diverse resource requirements. We show that our approach results in timely processing of Nowcast workflows under different infrastructure configurations and network conditions. We also show how workflow task clustering choices affect throughput of an ensemble of Nowcast workflows with improved turnaround times. Additionally, we find that using our network-centric platform powered by advanced layer2 networking techniques results in faster, more reliable data throughput, makes cloud resources easier to provision, and the workflows easier to configure for operational use and automation.

Science reproducibility is a cornerstone feature in scientific workflows. In most cases, this has been implemented as a way to exactly reproduce the computational steps taken to reach the final results. While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints and operating system name and versions. Though this may be sufficient for reproducibility in the short term, systems evolve and are replaced over time, breaking the underlying workflow reproducibility. A natural solution to this problem is containers, as they are well defined, have a lifetime independent of the underlying system, and can be user-controlled so that they can provide custom environments if needed. This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows. Further, this paper explores how the Pegasus Workflow Management System implements container support to address such challenges.

As the CASA DFW Urban Testbed has evolved from a small, Doppler radar network to an operational system providing user decision support, multi-sensor data, images thereof, and a real time alerting mechanism, it has become necessary to make use of the Compute Cloud to efficiently process and classify products in a timely and cost effective manner. The Global Environment for Network Innovations (GENI), was selected for this purpose and this manuscript shall describe how compute resources are managed to reduce idle time, how networks are configured dynamically to maximize performance, and how diverse products are accumulated and processed to serve the diverse user base in the DFW metroplex. We find that the GENI cloud offers superior networking, resource acquisition speed than several commodity clouds, and hardware performance on par.

High resolution, networks of X-band radars can improve severe weather warning operations by observing the lower troposphere at very high spatiotemporal resolution. X-band networks provide unique information on storm features that complement existing radars such as NEXRAD and TDWR. In this paper, we examine the warning benefits of these small radars by looking at the performance of a network of 7 X-band CASA radars in the Dallas Fort Worth Metroplex, linked to real-time product generation and decision-making. By evaluating two severe weather episodes, a squall line and a mesoscale convective system, we begin to identify the strengths and weaknesses of the X-band radar networks, and propose future benefits to warning decision making.

Large-scale wireless mesh networks, like the ones used as cellular back-haul, operate under circumstances, where individual links are affected by weather conditions. Reliability requirements in wireless mesh networks necessitate the ability to reconfigure the network in the face of changing environmental conditions. In this paper, we present a predictive routing protocol for wireless mesh networks, which operate at millimeter-wave bands with directional links, that uses in-network parameter prediction to make the network adaptive, as opposed to using meteorological weather information from external sources, such as weather radars. We validate our approach through simulations based on real-world weather events, observed through a network of weather radars, and comparisons with approaches that do not make use of predictions but may use the link quality as a parameter in routing decision making. Our results show that our link quality-based predictive approach can achieve throughput performance that is almost 8% better than a link quality-based routing algorithm that does not use prediction for the real weather scenario we use for our simulations.

Compute cloud platforms pay-as-you-use model suits applications which require resources sporadically. Severe weather detection and prediction is one such application. Since severe weather events are rare, dedicating servers for such application wastes resources. In this paper, we present the feasibility of using commercial cloud services for severe weather detection and prediction. We show that commercial cloud services provide the required network capability to perform the real-time operation of weather detection and prediction from the radars to the cloud service instance. We automate the process of weather prediction on the cloud based on the results of our weather detection algorithms.

Since today's weather forecasts only cover large regions every few hours, their use in severe weather is limited. In this paper, we present CloudCast, an application that provides short-term weather forecasts depending on users current location. Since severe weather is rare, CloudCast leverages pay-as-you-go cloud platforms to eliminate dedicated computing infrastructure. CloudCast has two components: 1) an architecture linking weather radars to cloud resources, and 2) a Nowcasting algorithm for generating accurate short-term weather forecasts. We study CloudCast's design space, which requires significant data staging to the cloud. Our results indicate that serial transfers achieve tolerable throughput, while parallel transfers represent a bottleneck for real-time mobile Nowcasting. We also analyze forecast accuracy and show high accuracy for ten minutes in the future. Finally, we execute CloudCast live using an on-campus radar, and show that it delivers a 15-minute Nowcast to a mobile client in less than 2 minutes after data sampling started.

Dedicating high-end servers for executing scientific applications that run intermittently, such as severe weather detection or generalized weather forecasting, wastes resources. While the Infrastructure-as-a-Service (IaaS) model used by today's cloud platforms is well-suited for the bursty computational demands of these applications, it is unclear if the network capabilities of today's cloud platforms are sufficient. In this paper, we analyze the networking capabilities of multiple commercial (Amazon's EC2 and Rackspace) and research (GENICloud and ExoGENI cloud) platforms in the context of a Nowcasting application, a forecasting algorithm for highly accurate, near-term, e.g., 5-20 minutes, weather predictions. The application has both computational and network requirements. While it executes rarely, whenever severe weather approaches, it benefits from an IaaS model; However, since its results are time-critical, enough bandwidth must be available to transmit radar data to cloud platforms before it becomes stale. We conduct network capacity measurements between radar sites and cloud platforms throughout the country. Our results indicate that ExoGENI cloud performs the best for both serial and parallel data transfer with an average throughput of 110.22 Mbps and 17.2 Mbps, respectively. We also found that the cloud services perform better in the distributed data transfer case, where a subset of nodes transmit data in parallel to a cloud instance. Ultimately, we conclude that commercial and research clouds are capable of providing sufficient bandwidth for our real-time Nowcasting application.

IP1 is a prototype CASA radar sensor network located in southwestern Oklahoma whose goal is to detect severe weather in the lower part of the atmosphere. At the center of this system's control loop is its Meteorological Command and Control (MC&C). In this paper, we presented the overall control architecture for the IP1 network and highlight new features that have recently been added to the MC&C. We also present an analysis of the MC&C performance based on measurement data from a 5-day operation period. In addition, we introduce a distributed version of the MC&C.

In this paper, we focus on a network of rich sensors that are geographically distributed and argue that the design of such networks poses very different challenges from traditional mote-class sensor network design. We identify the need to handle the diverse requirements of multiple users to be a major design challenge, and propose a utility-driven approach to maximize data sharing across users while judiciously using limited network and computational resources. Our utility-driven architecture addresses three key challenges for such rich multi-user sensor networks: how to define utility functions for networks with data sharing among end-users, how to compress and prioritize data transmissions according to its importance to end-users, and how to gracefully degrade end-user utility in the presence of bandwidth fluctuations. We instantiate this architecture in the context of geographically distributed wireless radar sensor networks for weather, and present results from an implementation of our system on a multi-hop wireless mesh network that uses real radar data with real end-user applications. Our results demonstrate that our progressive compression and transmission approach achieves an order of magnitude improvement in application utility over existing utility-agnostic non-progressive approaches, while also scaling better with the number of nodes in the network.

The emerging of rich sensor networks poses very different design challenges from traditional "mote-class" sensor networks. One important challenge is that these networks are designed to handle the diverse requirements of multiple users. In this work, we demonstrate how multiple end user needs are handled in rich sensor networks using a utility-driven architecture. We instantiate this architecture in the context of geographically distributed wireless radar sensor networks for weather, and demonstrate the real-time operation of the prototype on a radar testbed in Okalahoma.