Artificial intelligence (AI) has the potential to accelerate scientific discovery, yet many domain scientists face significant barriers when attempting to incorporate AI into their research. These challenges include a lack of formal training, limited access to computational resources, difficulty selecting appropriate models, and fragmented workflows. Through interviews with eleven postdoctoral researchers across diverse scientific fields at UC San Diego, we identify common pain points and recurring questions that arise throughout the scientific research pipeline. In response, we propose a conceptual framework, called Agents of Intelligence: modular, context-aware AI agents designed to assist researchers with tasks such as model recommendation, data preparation, infrastructure coordination, and interpretation of results. By embedding AI expertise into composable workflows, these agents aim to make AI more accessible, systematic, and transferable. We describe how this framework can be instantiated within the National Data Platform (NDP) and provide a testbed for real-world integration and evaluation.

Understanding spatiotemporal relationships in environmental monitoring is crucial for effective decision making and analysis. In this work, we leverage knowledge graphs (KGs) to model such relationships using measurements collected from field sensors. Specifically, we present datasets from a prescribed fire at Sedgwick Reserve, Santa Barbara, and introduce an ontology that defines key relationships between entities in these datasets. To facilitate data integration and analysis, we developed an ingestion pipeline that structures the information into a KG and facilitates querying along spatial and temporal edges. We evaluate our implementation with two tasks, the ability for spatial relationships to improve vegetation metric prediction, and the use of the inferred relationships between carbon emission data and vegetation metrics to estimate prescribed fire outcomes. Our approach highlights the potential of knowledge graphs in environmental data management, offering a scalable and interactive framework for analysis and decision support.

Wildfires are becoming increasingly destructive and costly each year, affecting lives, damaging infrastructure, and degrading ecosystems. To address this growing threat, the fire and land management community need smarter, data-driven tools to understand the landscape and plan their essential prescribed burns that reduce hazardous fuels. With the ultimate goal to minimize the devastation of wildfires by enabling proactive and data-driven fuel management at landscape scale, this paper presents an approach that builds heterogeneous remote sensing data into a temporal-spatial knowledge graph, then queries it with a Large Language Model (LLM) based agent, providing a natural language interface to an extensive system of granular landscape knowledge and metrics. We demonstrate how we build knowledge graphs from LiDAR-derived vegetation metrics using GraphDB enabling precise location and time-based insights. We present a user-facing system intended to respond to queries about the effects of prescribed burns over time. Built around an LLM Agent (e.g., OpenAI, LLaMA) orchestrated with LangChain and LangGraph, the system allows users to interact with complex fire and fuel data through a natural language chat interface. It also includes a web search tool for retrieving external fire-related content to enrich responses. While this work operates as a standalone knowledge system, it was first envisioned as a method of generating conversational narratives while navigating a virtual 3D forest environment in our prototype immersive visualization application called Immersive Forest.

In recent years, digital twins have emerged as an advanced data-driven technology for modeling and monitoring complex physical systems to inform actionable decision-making. One area where digital twins can be especially impactful is urgent computing applications, such as natural disasters and hazards, where timely and reliable decision support is needed. However, it is challenging to streamline dynamic data with modeling and simulation methods for urgent computing applications due to the use of heterogeneous multimodal data sources, Internet of Things (IoT) networks, and high-performance computing (HPC) systems to power edge-to-cloud capabilities. In this work, we describe a reference architecture that uses federation and dynamic composability to power digital twins for complex physical systems. We demonstrate the usefulness of our approach with the Firemap architecture on WIFIRE, an end-to-end cyberinfrastructure for real-time and data-driven simulation, prediction, and visualization of wildfire behavior.

Distributed computing systems are essential for meeting the demands of modern HPC applications, yet transitioning from single-system to distributed environments presents significant challenges. Resource misallocation in shared systems can lead to resource contention, system instability, degraded performance, priority inversion, inefficient utilization, increased latency, and environmental impact. We present BanditWare, an online recommendation system that dynamically selects the most suitable hardware for applications using a contextual multi-armed bandit algorithm. We evaluated BanditWare on two workflow applications: BurnPro3D (a web-based platform for fire science), and a matrix multiplication application. Designed for seamless integration with the National Data Platform (NDP), BanditWare enables users of all experience levels to optimize resource allocation efficiently.

This project aims at developing an AI system to provide a reliable assessment of the structural damage caused by wildfires in the first burn period. Our approach uses multimodal data, including multispectral aerial images, historical post-fire damage assessment data, and building footprints, to create an association between damage data and structure footprints. We use these associations to generate features and use machine learning methods to assess the level of damage to structures. The resulting AI-driven system can be used to provide wildfire-induced structural damage assessments in near-real-time using only aerial images for future fires. We provide damage assessment results on several megafires in California to demonstrate the applicability of our approach to real wildfire scenarios.

In recent years, frequent and highly destructive megafires become one of the biggest climate-induced disasters. Fire behavior models using data from many emerging sources can inform decision support tools to respond to and mitigate such megafires. Emerging edge sensing and computing technologies within the fire environment can enhance the speed, reliability, and efficiency of wildland fire management, leading to better prevention, faster response times, and more effective mitigation of fire-related disasters. However, a unified system that streamlines the integration of edge technology advances within fire science and management workflows is needed. This paper presents the design and demonstrated case studies of the WIFIRE Edge Platform that facilitates the integration of sensing and AI capabilities at the edge. The initial attack and prescribed burn concept scenarios are described, highlighting the sensor deployment and utilization at the fire front.

“CCGRID will act with responsibility as its primary consideration; with equity, diversity, and inclusion as its central goals.” from the CCGRID 2023 web site [1]

eScience is now an example field for conducting science using advanced computational methodologies, middle-ware and workflows. With the ever-increasing data intensity in various scientific disciplines and the opportunities provided by novel artificial intelligence (AI) and computing techniques, there is a pressing need for eScience to adapt and develop holistic approaches for AI-integrated science and interdisciplinary collaboration between science, AI and computing experts. This paper discusses the transformative shift needed in the conduct and tools of eScience from traditional workflows, often linear, pre-defined to coordinate task execution and individually developed, to integrated “teamflows”, a term coined to encapsulate methods for effective team collaboration and communication, dynamic data and service integration, and use-inspired thinking. In the AI in science era of scientific investigation, this “future of eScience” special track paper: 1) emphasizes the vital importance of eScience to the success of collaborative AI in science teams; 2) recites some requirements for open data and services for AI-enhanced workflows, team science and incentive structures to create a more inclusive and efficient discovery environment; and 3) discusses impact as a measure of success in eScience.

Reproducibility plays an essential role in scientific research to ensure accuracy and serves as a foundation for future advancements. Scientific reproducibility becomes particularly challenging when dealing with vast amounts of input files that change hands or move across different laboratories or organizations. Preserving the provenance of data files ensures critical information about the originality of data files is captured to support the reproducibility of scientific research. The paper focuses on capturing and verifying input and output data file provenance using the principles of blockchain. The technique stores the hashes of data files in a database along with user and workflow information. It allows the workflow to verify the data against the hashes at any point. The method is demonstrated using Parflow, a Hydrologic model, as a proof-of-concept.

Wildland fire modeling tools can ingest high resolution 3D vegetation models as inputs. However, data used to build the surface fuels in these models is often at a 30-meter resolution, which does not necessarily provide sufficient detail for accurate modeling of fires. Terrestrial laser scans are increasingly being used to collect detailed vegetation data that could be integrated with new approaches to fuel and fire modeling, but manual segmentation of scans is not scalable beyond a small number of scans. There is a need to automatically segment these high resolution point clouds as they are collected in the field, such that they may be leveraged by fuel and fire models for wildland fire response and mitigation and other applied climate science. This paper summarizes our early work on a labeling, visualization and machine learning pipeline for detailed segmentation of fuels. Specific contributions are: (1) a labeling approach involving 3 dimensional segmentation of point clouds using a point cloud processing engine; (2) a visualization approach using a computer graphics engine; and (3) early results from a deep learning modeling approach for fuel segmentation by category (live and dead) and size class (1, 10, 100 and 1000 hour fuels).

Open and equitable access to scientific data is essential to addressing important scientific and societal grand challenges, and to research enterprise more broadly. This paper discusses the importance and urgency of open and equitable data access, and explores the barriers and challenges to such access. It then introduces the vision and architecture of the National Data Platform, a recently launched project aimed at catalyzing an open, equitable and extensible data ecosystem.

Long-standing fire suppression policies, global warming and human influence at the urban-wildland interface are fueling a global wildfire crisis. Prescribed burns are increasingly being recognized as an essential procedure to reduce fuel (biomass) overgrowth and mitigate the size and severity of uncontrolled wildfires. Next-generation three-dimensional (3D) fire behavior simulations, which can help land managers to identify risks and improve planning for successful prescribed burns, depend on accurate 3D fuel structure models. We introduce TrueTrees, a workflow that integrates tree-level observations from airborne lidar surveys into FastFuels 3D fuel models. The workflow is optimized and distributed to allow processing of point cloud data for typical burn units within minutes. TrueTrees is implemented into a prototype of BurnPro3D, a user-friendly science-driven decision support platform for prescribed burn planners.

The FAIR and CARE data principles are critical to ensuring widespread and equitable access to open data. They provide guidelines for what should be contained in metadata and how certain types of data should be handled. This study examines how large, collective data hubs such as the WIFIRE Data Commons can implement the FAIR and CARE data principles through evaluating the datasets hosted on the platform. An automation pipeline was developed to check for specified criteria in these principles, allowing fast integration of the principles on a large scale data hub. This pipeline can be expanded to check for all FAIR and CARE criteria, and similar pipelines can be created for a variety of other data hubs. Automating for the FAIR and CARE principles will help simplify organization of open data, allowing for a greater expansion of open science.

Timely prediction of debris flow probabilities in areas impacted by wildfires is crucial to mitigate public exposure to this hazard during post-fire rainstorms. This paper presents a machine learning approach to amend an existing dataset of post-fire debris flow events with additional features reflecting existing vegetation type and geology, and train traditional and deep learning methods on a randomly selected subset of the data. The developed methods achieve AUC (area under the receiver operational characteristic curve) values of 0.93 (random forest) and 0.92 (neural network) on the test set, representing a significant improvement over a logistic regression model currently used (AUC 0.79). The paper also overviews a distributed, Kubernetesbased big data processing pipeline to efficiently retrieve features in areas impacted by new fires, and deploy the methods for real-time prediction of debris flow hazards.

Influenced by the advances in data and computing, the scientific practice increasingly involves machine learning and artificial intelligence driven methods which requires specialized capabilities at the system-, science- and service-level in addition to the conventional large-capacity supercomputing approaches. The latest distributed architectures built around the composability of data-centric applications led to the emergence of a new ecosystem for container coordination and integration. However, there is still a divide between the application development pipelines of existing supercomputing environments, and these new dynamic environments that disaggregate fluid resource pools through accessible, portable and re-programmable interfaces. New approaches for dynamic composability of heterogeneous systems are needed to further advance the data-driven scientific practice for the purpose of more efficient computing and usable tools for specific scientific domains. In this paper, we present a novel approach for using composable systems in the intersection between scientific computing, artificial intelligence (AI), and remote sensing domain. We describe the architecture of a first working example of a composable infrastructure that federates Expanse, an NSF-funded supercomputer, with Nautilus, a Kubernetes-based GPU geo-distributed cluster. We also summarize a case study in wildfire modeling, that demonstrates the application of this new infrastructure in scientific workflows: a composed system that bridges the insights from edge sensing, AI and computing capabilities with a physics-driven simulation.

Increasing social acceptance of prescribed burns is an important element of ramping up these controlled burns to the scale required to effectively mitigate destructive wildfires through reduction of excessive fire fuel loads. As part of a Design Challenge, students created concept designs for physical or virtual installations that would increase public understanding and acceptance of prescribed burns as an important tool for ending devastating megafires. The proposals defined how the public would interact with the installation and the learning goals for participants. This poster provides an overview of the virtual reality (VR) pipeline created to develop working prototypes of the immersive experiences and VR games that were proposed by the finalists in the design challenge.

Knowledge networks/graphs provide a powerful approach for data discovery, integration, and reuse. The NSF's new Convergence Accelerator program, which focuses on transitioning research to practice and translational research, announced Track A on the Open Knowledge Network (OKN). The program calls for multidisciplinary and multi-sector teams to work together to build a cooperative and shared open knowledge network infrastructure to drive innovation across science, engineering, and humanities. This workshop aims to invite researchers, practitioners, and the general public to brainstorm the ideas related to OKN, collaboratively build KGs for different domains or applications, develop AI algorithms to provide intelligent services based on OKN, and discuss the social and economic implications related to OKN.

A key takeaway from the COVID-19 crisis is the need for scalable methods and systems for ingestion of big data related to the disease, such as models of the virus, health surveys, and social data, and the ability to integrate and analyze the ingested data rapidly. One specific example is the use of the Internet of Things and wearables (i.e., the Oura ring) to collect large-scale individualized data (e.g., temperature and heart rate) continuously and to create personalized baselines for detection of disease symptoms. Individualized data, when collected, has great potential to be linked with other datasets making it possible to combine individual and societal scale models for further understanding the disease. However, the volume and variability of such data require novel big data approaches to be developed as infrastructure for scalable use. This paper presents the data pipeline and big data infrastructure for the TemPredict project, which, to the best of our knowledge, is the largest public effort to gather continuous physiological data for time-series analysis. This effort unifies data ingestion with the development of a novel end-to-end cyberinfrastructure to enable the curation, cleaning, alignment, sketching, and passing of the data, in a secure manner, by the researchers making use of the ingested data for their COVID-19 detection algorithm development efforts. We present the challenges, the closed-loop data pipelines, and the secure infrastructure to support the development of time-sensitive algorithms for alerting individuals based on physiological predictors illness, enabling early intervention.

Summary form only given. The complete presentation was not made available for publication as part of the conference proceedings. Cyberinfrastructure is everywhere in diverse forms. From IoT to extreme scale computing, data and computing has never been as distributed with potential for real-time integration via fast networking and container management. The growth of new processors over the last decade including GPUs, FPGAs, and edge accelerators opened the way to a diverse set of applications using machine learning on top of distributed nontraditional hardware. The common theme to these applications, mostly composed of artificial intelligence (AI) workloads, is their need to run in specialized environments for reasons such as on demand or 24×7 nature of the tasks they are performing, and difficulties regarding their portability, latency, privacy and performance optimization. In many data-driven scientific applications there is a need for integration of these AI-workloads with traditional high-throughput computing (HTC) or high-performance computing (HPC) tasks for AI-integrated science. This talk will discuss example AI-integrated applications, describe some of the new systems that enabled these applications, and overview our recent research to enable composable applications including an application development methodology, intelligent middleware and workflow composition.

The Neuroscience domain stands out from the field of sciences for its dependence on the study and characterization of complex, intertwining structures. Understanding the complexity of the brain has led to widespread advances in the structure of large-scale computing resources and the design of artificially intelligent analysis systems. However, the scale of problems and data generated continues to grow and outpace the standards and practices of neuroscience. In this paper, we present an automated neuroscience reconstruction framework, called NeuroKube, for large-scale processing and labeling of neuroimage volumes. Automated labels are generated through a machine-learning (ML) workflow, with data-intensive steps feeding through multiple GPU stages and distributed data locations leveraging autoscalable cloud-native deployments on a multi-institution Kubernetes system. Leading-edge hardware and storage empower multiple stages of machine-learning, GPU-accelerated solutions. This demonstrates an abstract approach to allocating the resources and algorithms needed to elucidate the highly complex structures of the brain. We summarize an integrated gateway architecture, and a scalable workflow-driven segmentation and reconstruction environment that brings together image big data with state-of-the-art, extensible machine learning methods.

High-resolution satellite imagery is a rich source of data applicable to a variety of domains, ranging from demo-graphics and land use to agriculture and hazard assessment. We have developed an end-to-end analysis pipeline that uses deep learning and unsupervised learning to process high-resolution satellite imagery and have applied it to various applications in previous work. As high-resolution satellite imagery is large-volume data, scalability is important to be able to analyze data from large geographical areas. To add scalability to our process, we converted our original pipeline, implemented using the Caffe deep learning library and the Python machine learning library Scikit-Learn, to other platforms that make use of distributed computation. Specifically, to add scalability, we use Keras for deep learning, and evaluate two different distributed platforms, Spark and Dask, for unsupervised learning. We report on results in scaling up our satellite analysis pipeline.

In summer 2017, close to one million Rohingya, an ethnic minority group in Myanmar, have fled to Bangladesh due to the persecution of Muslims. This large influx of refugees has resided around existing refugee camps. Because of this dramatic expansion, the newly established Kutupalong-Balukhali expansion site lacked basic infrastructure and public service. While Non-Governmental Organizations (NGOs) such as Refugee Relief and Repatriation Commissioner (RRCC) conducted a series of counting exercises to understand the demographics of refugees, our understanding of camp formation is still limited. Since the household type survey is time-consuming and does not entail geo-information, we propose to use a combination of high-resolution satellite imagery and machine learning (ML) techniques to assess the spatiotemporal dynamics of the refugee camp. Four Very-High Resolution (VHR) images (i.e., World View-2) are analyze to compare the camp pre-and post-influx. Using deep learning and unsupervised learning, we organized the satellite image tiles of a given region into geographically relevant categories. Specifically, we used a pre-trained convolutional neural network (CNN) to extract features from the image tiles, followed by cluster analysis to segment the extracted features into similar groups. Our results show that the size of the built-up area increased significantly from 0.4 km² in January 2016 and 1.5 km² in May 2017 to 8.9 km² in December 2017 and 9.5 km² in February 2018. Through the benefits of unsupervised machine learning, we further detected the densification of the refugee camp over time and were able to display its heterogeneous structure. The developed method is scalable and applicable to rapidly expanding settlements across various regions. And thus a useful tool to enhance our understanding of the structure of refugee camps, which enables us to allocate resources for humanitarian needs to the most vulnerable populations.

Founded in 2018, the Halıcıoğlu Data Science Institute (HDSI) is a significant new organization on the UC San Diego campus. As part of their pedagogical processes, HDSI faculty desired a way to store and share student capstone projects with future cohorts, so students could easily access reusable, raw datasets and analytical workflows, with the potential to expand on work done by previous cohorts. The UC San Diego Library has been managing an institutional data repository for over a decade, with established ingest workflows and tools. The Library's Digital Collections accommodates collections of objects, complex organization of data and metadata, batch ingest options, and deposit of large datasets. Curators review data submissions and work with data depositors to enrich collections with descriptive metadata, controlled vocabularies, and persistent identifiers. HDSI faculty are working with the Library's Data Science Librarian and members of the Library's Research Data Curation program to design a custom workflow to facilitate ingest of an entire class worth of projects within a relatively quick time frame and have students prepare their materials with minimal one-on-one instruction from Library staff. The first test of this workflow ingests project materials from the Data Science & Engineering Master of Advanced Study capstone course into the Library's digital repository and makes the materials available using community-standard discoverability tools. This process represents the first ingestion of data- and computationally-intensive student projects into the repository and is intended to provide a template for a scalable workflow to accommodate other courses, ultimately creating a series of course data collections to support teaching and learning.

In 2016, a team of earth scientists directly engaged a team of computer scientists to identify cyberinfrastructure (CI) approaches that would speed up an earth science workflow. This paper describes the evolution of that workflow as the two teams bridged CI and an image segmentation algorithm to do large scale earth science research. The Pacific Research Platform (PRP) and The Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI) resources were used to significantly decreased the earth science workflow's wall-clock time from 19.5 days to 53 minutes. The improvement in wall-clock time comes from the use of network appliances, improved image segmentation, deployment of a containerized workflow, and the increase in CI experience and training for the earth scientists. This paper presents a description of the evolving innovations used to improve the workflow, bottlenecks identified within each workflow version, and improvements made within each version of the workflow, over a three-year time period.

Author(s): Rule, Adam; Birmingham, Amanda; Zuniga, Cristal; Altintas, Ilkay; Huang, Shih-Cheng; Knight, Rob; Moshiri, Niema; Nguyen, Mai H; Rosenthal, Sara Brin; Pérez, Fernando; Rose, Peter W | Editor(s): Lewitter, Fran

Multi-scale computational modeling is a major branch of computational biology as evidenced by the US federal interagency Multi-Scale Modeling Consortium and major international projects. It invariably involves specific and detailed sequences of data analysis and simulation, often with multiple tools and datasets, and the community recognizes improved modularity, reuse, reproducibility, portability and scalability as critical unmet needs in this area. Scientific workflows are a well-recognized strategy for addressing these needs in scientific computing. While there are good examples if the use of scientific workflows in bioinformatics, medical informatics, biomedical imaging and data analysis, there are fewer examples in multi-scale computational modeling in general and cardiac electrophysiology in particular. Cardiac electrophysiology simulation is a mature area of multi-scale computational biology that serves as an excellent use case for developing and testing new scientific workflows. In this article, we develop, describe and test a computational workflow that serves as a proof of concept of a platform for the robust integration and implementation of a reusable and reproducible multi-scale cardiac cell and tissue model that is expandable, modular and portable. The workflow described leverages Python and Kepler-Python actor for plotting and pre/post-processing. During all stages of the workflow design, we rely on freely available open-source tools, to make our workflow freely usable by scientists.

In the history of Computer Science, the act of `delegation' has been the greatest multiplier of society's problem-solving ability. A scientist working on detecting anomalies in a phenomenon, does not need to re-invent matrix multiplication techniques to solve her problem. Scientific workflows provide ultimate `delegation' mechanism - where a domain scientist can completely forget the specifics of `how' her program will execute on a large cluster in an efficient and cost-effective manner and can instead focus on the mathematical formulation and theoretical robustness of her solution. We present here an approach that directly aims to make the execution of Scientific Workflows more reliable, robust and efficient. We aim that the work presented in this paper will propel the larger effort, from the scientific workflow community, of making scientific workflow execution as simple, efficient and robust as a JOIN operation in a modern database. Specifically, we apply Deep Learning techniques to develop a mechanism that forecasts the final state (success or failure) of a dynamic job in a large-scale particle physics experiment, with minimal data gathering, and as early as possible in job's life cycle. The key advantage of having a predictive mechanism to identify and anticipate failure-prone jobs is the potential for designing intelligent Fault Tolerance mechanisms to handle anomalous events. We achieve a 14% improvement in computational resources utilization, and an overall classification accuracy of 85% on real tasks executed in a High Energy Physics Computing workflow. To the best of our knowledge, this is the most exhaustive and first of its kind study of neural network architectures in context of a real-dataset profiled from a large-scale scientific workflow.

Land cover classification analysis from satellite imagery is important for monitoring change in ecosystems and urban growth over time. However, the land cover classifications that are widely available in the United States are generated at a low spatial and temporal resolution, so that the spatial distribution between vegetation and urban areas in the wildland urban interface is difficult to measure. High spatial and temporal resolution analysis is essential for understanding and managing changing environments in these regions. This paper describes an end to end satellite data ingestion and analysis pipeline using deep learning on high resolution satellite imagery for generating pixel-based land cover classification.

The left ventricle (LV) is the largest chamber in the heart and plays a critical role in cardiac function. Noninvasive cardiac imaging modalities (e.g., cardiac magnetic resonance (CMR), transesophageal echocardiography (TEE), and computed tomography (CT)) are commonly used to study LV size and function in addition to other cardiac structural aspects such as valvular disease, and are invaluable tools for the diagnosis and management of heart disease. However, the process of analyzing cardiac images is time-consuming and labor-intensive. Automatic LV segmentation and volume estimation from cardiac images are thus essential in providing efficient and consistent analysis. We discuss findings from our investigation into different techniques for processing and analyzing CMR images and present the methods giving best performance in an end -to -end analytics pipeline for LV segmentation and volume estimation. This pipeline can serve as an initial step towards analyzing CMR at scale to aid in non-invasive cardiac disease detection.

Southern California's dry climate and fire-prone vegetation make the area vulnerable to extreme wildfire conditions. These conditions are exacerbated by Santa Ana weather patterns, which are characterized by very low humidity and gusty winds blowing in from the deserts. We present an approach using unsupervised learning to model and detect Santa Ana conditions based on sensor measurements from weather stations. Our approach uses cluster analysis to capture weather patterns specific to the region surrounding each weather station. A method is provided to automatically determine the Santa Ana cluster for each cluster model using dynamic, data-driven criteria. The resulting cluster models are applied to real-time sensor measurements to provide location-specific and time-specific detection of Santa Ana conditions. The Spark distributed platform is leveraged to scale the system to large datasets from multiple weather stations, and the Kepler workflow system is used to provide a GUI-based, easy-to-use interface to the underlying system. Results of testing our approach on an existing network of weather stations are presented. Our scalability experiment shows that the approach can process up to one million live sensor measurements in less than one minute on one machine. The proposed system can be used to aid in wildfire management and prevention by focusing firefighting efforts on regions with increased wildfire risks.

High resolution satellite imagery is a growing source of data with potential applications in many diverse domains. Efficient large scale analysis of this rich data can lead to unprecedented discoveries with societal impact. We present a new framework for organizing collections of satellite images into demographically relevant categories using unsupervised learning techniques. Our framework first extracts features using pre-trained Convolutional Neural Networks from tiles of high resolution satellite images of a city. The k-means algorithm is then applied to these features to organize images into visually similar groups. The resulting clustered images are validated using demographic data. The cluster model is then applied to six different cities around the world to test the transferability of our methods. Finally, the discovered image clusters are visualized in our customized web interface to enable demographers, social scientists, and economists to understand the organization of a city.

The provisioning and exploitation at a global scale of environmental and geophysical data requires advanced automation and governance mechanisms that enable (meta)data interoperability but also the exchange of formalized scientific concepts and methods. In this paper we introduce recent efforts in such direction, based on scientific workflows and research objects as enablers of such vision. The former enable the integration of different web services in a higher-level data processing artifact while the latter enhances governance around data product validation and consistency, result reproducibility and credit to the principal investigators and data providers. This paper provides a concise overview of our project, current status and next steps.

Distributed computing platforms provide a robust mechanism to perform large-scale computations by splitting the task and data among multiple locations, possibly located thousands of miles apart geographically. Although such distribution of resources can lead to benefits, it also comes with its associated problems such as rampant duplication of file transfers increasing congestion, long job completion times, unexpected site crashing, suboptimal data transfer rates, unpredictable reliability in a time range, and suboptimal usage of storage elements. In addition, each sub-system becomes a potential failure node that can trigger system wide disruptions. In this vision paper, we outline our approach to leveraging Deep Learning algorithms to discover solutions to unique problems that arise in a system with computational infrastructure that is spread over a wide area. The presented vision, motivated by a real scientific use case from Belle II experiments, is to develop multilayer neural networks to tackle forecasting, anomaly detection and optimization challenges in a complex and distributed data movement environment. Through this vision based on Deep Learning principles, we aim to achieve reduced congestion events, faster file transfer rates, and enhanced site reliability.

Today large amount of data is generated by cities. Many of the datasets are openly available and are contributed by different sectors, government bodies and institutions. The new data can affect our understanding of the issues faced by cities and can support evidence based policies. However usage of data is limited due to difficulty in assimilating data from different sources. Open datasets often lack uniform structure which limits its analysis using traditional database systems. In this paper we present Citadel, a data hub for cities. Citadel's goal is to support end to end knowledge discovery cyber-infrastructure for effective analysis and policy support. Citadel is designed to ingest large amount of heterogeneous data and supports multiple use cases by encouraging data sharing in cities. Our poster presents the proposed features, architecture, implementation details and initial results.

Advances in satellite imagery presents unprecedented opportunities for understanding natural and social phenomena at global and regional scales. Although the field of satellite remote sensing has evaluated imperative questions to human and environmental sustainability, scaling those techniques to very high spatial resolutions at regional scales remains a challenge. Satellite imagery is now more accessible with greater spatial, spectral and temporal resolution creating a data bottleneck in identifying the content of images. Because satellite images are unlabeled, unsupervised methods allow us to organize images into coherent groups or clusters. However, the performance of unsupervised methods, like all other machine learning methods, depends on features. Recent studies using features from pre-trained networks have shown promise for learning in new datasets. This suggests that features from pre-trained networks can be used for learning in temporally and spatially dynamic data sources such as satellite imagery. It is not clear, however, which features from which layer and network architecture should be used for learning new tasks. In this paper, we present an approach to evaluate the transferability of features from pre-trained Deep Convolutional Neural Networks for satellite imagery. We explore and evaluate different features and feature combinations extracted from various deep network architectures, and systematically evaluate over 2,000 network-layer combinations. In addition, we test the transferability of our engineered features and learned features from an unlabeled dataset to a different labeled dataset. Our feature engineering and learning are done on the unlabeled Draper Satellite Chronology dataset, and we test on the labeled UC Merced Land dataset to achieve near state-of-the-art classification results. These results suggest that even without any or minimal training, these networks can generalize well to other datasets. This method could be useful in the task of clustering unlabeled images and other unsupervised machine learning tasks.

Santa Ana conditions are hot, dry, windy weather conditions that can greatly increase the dangers of wildfires in southern California. We present a machine learning approach to detect Santa Ana conditions based on sensor measurements from weather stations. Cluster analysis is performed on historical weather data to build models to identify Santa Ana patterns. A separate model is built using data from each weather station to capture the patterns specific to the microclimate of each region. Real-time sensor data from a weather station can then be processed to determine if the region surrounding that station is experiencing Santa Ana conditions. Results can be used as a warning system to focus firefighting efforts on regions with increased wildfire risks. Through the use of the Kepler workflow system and distributed computing with Spark, data from several weather stations can be processed in parallel using a scalable clustering algorithm, allowing our approach to scale to large datasets from multiple weather stations.

Today's dynamic computing deployment for commercial and scientific applications is propelling us to an era where minor inefficiencies can snowball into significant performance and operational bottlenecks. Data center operations is increasingly relying on sensors based control systems for key decision insights. The increased sampling frequencies, cheaper storage costs and prolific deployment of sensors is producing massive volumes of operational data. However, there is a lag between rapid development of analytical techniques and its widespread practical deployment. We present empirical evidence of the potential carried by analytical techniques for operations management in computing and data centers. Using Machine Learning modeling techniques on data from a real instrumented cluster, we demonstrate that predictive modeling on operational sensor data can directly reduce systems operations monitoring costs and improve system reliability.

Ability to track provenance is a key feature of scientific workflows to support data lineage and reproducibility. The challenges that are introduced by the volume, variety and velocity of Big Data, also pose related challenges for provenance and quality of Big Data, defined as veracity. The increasing size and variety of distributed Big Data provenance information bring new technical challenges and opportunities throughout the provenance lifecycle including recording, querying, sharing and utilization. This paper discusses the challenges and opportunities of Big Data provenance related to the veracity of the datasets themselves and the provenance of the analytical processes that analyze these datasets. It also explains our current efforts towards tracking and utilizing Big Data provenance using workflows as a programming model to analyze Big Data.

Both improving the execution efficiency and reducing the execution cost are essential for scientific workflows in cloud environments. As many scientific workflow tasks become more data-intensive and computation-intensive, storing their outputs in cloud for reuse is a feasible way to achieve such objectives. Because data storage in cloud would increase the storage cost, though it might mean less computation time due to data reuse, it is important to determine the proportion of the output data sets of workflow tasks that should be stored in cloud. The paper explores caching provenance in cloud to enhance the smart re-run of Kepler workflows based on a near optimum data caching policy. Ant Colony System optimization is introduced to determine the near optimum data sets to be cached in cloud in order to improve the execution efficiency of future workflow re-run without increasing the total cost of workflow execution in cloud. Simulation and analysis show that the proposed approach is efficient.

BACKGROUND: Mandatory deposit of raw microarray data files for public access, prior to study publication, provides significant opportunities to conduct new bioinformatics analyses within and across multiple datasets. Analysis of raw microarray data files (e.g. Affymetrix CEL files) can be time consuming, complex, and requires fundamental computational and bioinformatics skills. The development of analytical workflows to automate these tasks simplifies the processing of, improves the efficiency of, and serves to standardize multiple and sequential analyses. Once installed, workflows facilitate the tedious steps required to run rapid intra- and inter-dataset comparisons. RESULTS: We developed a workflow to facilitate and standardize Meta-Analysis of Affymetrix Microarray Data analysis (MAAMD) in Kepler. Two freely available stand-alone software tools, R and AltAnalyze were embedded in MAAMD. The inputs of MAAMD are user-editable csv files, which contain sample information and parameters describing the locations of input files and required tools. MAAMD was tested by analyzing 4 different GEO datasets from mice and drosophila.MAAMD automates data downloading, data organization, data quality control assesment, differential gene expression analysis, clustering analysis, pathway visualization, gene-set enrichment analysis, and cross-species orthologous-gene comparisons. MAAMD was utilized to identify gene orthologues responding to hypoxia or hyperoxia in both mice and drosophila. The entire set of analyses for 4 datasets (34 total microarrays) finished in ~ one hour. CONCLUSIONS: MAAMD saves time, minimizes the required computer skills, and offers a standardized procedure for users to analyze microarray datasets and make new intra- and inter-dataset comparisons.

The Kepler scientific workflow system enables creation, execution and sharing of workflows across a broad range of scientific and engineering disciplines while also facilitating remote and distributed execution of workflows. In this paper, we present

To execute workflows on a compute cluster resource, workflow engines can work with cluster resource manager software to distribute jobs into compute nodes on the cluster. We discuss how to interact with traditional Oracle Grid Engine and recent Hadoop cluster resource managers using a dataflow-based scheduling approach to balance compute resource load for data-parallel workflow execution. Our experiments show that: 1) The presented approach can balance computational resource load well by interacting with the resource managers and provides good execution performance on both physical and virtual clusters, 2) Oracle Grid Engine outperforms Hadoop for CPU-intensive applications on small-scale clusters.

Data quality is an important component of modern scientific discovery. Many scientific discovery processes consume data from a diverse array of resources such as streaming sensor networks, web services, and databases. The validity of a scientific computation's results is highly dependent on the quality of these input data. Scientific workflow systems are being increasingly used to automate scientific computations by facilitating experiment design, data capture, integration, processing, and analysis. These workflows may execute in grid or cloud environments, and if the data produced during workflow execution is deemed unusable or low in quality, execution should stop to prevent wasting these valuable resources. We propose an approach in the Kepler scientific workflow system for monitoring data quality and demonstrate its use for oceanography and bioinformatics domains.

Over the last decade, workflows have been established as a mechanism for scientific developers to create simplified views of complex scientific processes. However, there is a need for a comprehensive system architecture to link scientific developers creating workflows with researchers launching workflows in large scale computing environments. We present the architecture for the CAMERA 2.0 Cyber infrastructure platform that provides a scaffold where workflows can be uploaded into the system, and user interface components for launching and viewing results are automatically generated. In CAMERA 2.0, scientific developers and metagenomics researchers seamlessly collaborate to (i) wrap data-analysis software applications and heterogeneous tools as Resource Oriented Architecture (ROA) components integrating them using scientific workflows; (ii) publish and run scientific workflows via dynamically generated uniform portal interfaces; (iii) map heterogeneous workflow products to provenance and CAMERA semantic database through a transformation component, to save output data resulting from workflow runs based on this mapping; (iv) record and visualize the provenance of all workflow run-related data and processes; and (v) conduct queries across multiple workflow executions and link these workflow executions to each other through data and provenance related to these runs. Furthermore, workflows added to CAMERA also have access to a variety of physical resources for computation and data management. Here, we demonstrate the usability of this framework with some of the developed metagenomics workflows.

This paper addresses the challenges of providing fault-tolerance in scientific workflow management. The specification and handling of faults in scientific workflows should be defined precisely in order to ensure the consistent execution against the process-specific requirements. We identified a number of typical failure patterns that occur in real-life scientific workflow executions. Following the intuitive recovery strategies that correspond to the identified patterns, we developed the methodologies that integrate recovery fragments into fault-prone scientific workflow models. Compared to the existing fault-tolerance mechanisms, the propositions reduce the effort of workflow designers by defining recovery fragments automatically. Furthermore, the developed framework implements the necessary mechanisms to capture the faults from the different layers of a scientific workflow management architecture. Experience indicates that the framework can be employed effectively to model, capture and tolerate the typical failure patterns that we identified.

Domain scientists synthesize different data and computing resources to solve their scientific problems. Making use of distributed execution within scientific workflows is a growing and promising way to achieve better execution performance and efficiency. This paper presents a high-level distributed execution framework, which is designed based on the distributed execution requirements identified within the Kepler community. It also discusses mechanisms to make the presented distributed execution framework easy-to-use, comprehensive, adaptable, extensible and efficient.

A challenge for Grid computing is the difficulty in developing software that is parallel, distributed and highly dynamic. Whilst there have been many general purpose mechanisms developed over the years, Grid programming still remains a low level, error prone task. Scientific workflow engines can double as programming environments, and allow a user to compose dasiavirtualpsila Grid applications from pre-existing components. Whilst existing workflow engines can specify arbitrary parallel programs, (where components use message passing) they are typically not effective with large and variable parallelism. Here we discuss dynamic dataflow, originally developed for parallel tagged dataflow architectures (TDAs), and show that these can be used for implementing Grid workflows. TDAs spawn parallel threads dynamically without additional programming. We have added TDAs to Kepler, and show that the system can orchestrate workflows that have large amounts of variable parallelism. We demonstrate the system using case studies in chemistry and in cardiac modelling.

Abstract Scientific workflows often benefit from or even require advanced modeling constructs, e.g. nesting of subworkflows, cycles for executing loops, data‐dependent routing, and pipelined execution. In such settings, an often overlooked aspect of provenance takes center stage: a suitable model of provenance (MoP) for scientific workflows should be based upon the underlying model of computation (MoC) used for executing the workflows. We can derive an adequate MoP from a MoC (such as Kahn's process networks) by taking into account the assumptions that a MoC entails, and by recording the observables which it affords. In this way, a MoP captures or at least better approximates ‘real’ data dependencies for workflows with advanced modeling constructs. As a specific instance, we elaborate on the Read–Write–ReSet model, a simple and flexible MoP suitable for a number of different MoCs. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract The first Provenance Challenge was set up in order to provide a forum for the community to understand the capabilities of different provenance systems and the expressiveness of their provenance representations. To this end, a functional magnetic resonance imaging workflow was defined, which participants had to either simulate or run in order to produce some provenance representation, from which a set of identified queries had to be implemented and executed. Sixteen teams responded to the challenge, and submitted their inputs. In this paper, we present the challenge workflow and queries, and summarize the participants' contributions. Copyright © 2007 John Wiley & Sons, Ltd.

Data Grid and Web Services are among the advanced computing technologies that are available to support scientists and scientific applications. We use Kepler scientific workflow system to integrate the two popular technologies for e-science applications. A prototype system for exploring species distribution patterns has been developed for demonstration purposes by using data grid resources and a similarity-based clustering Web service in conjunction with Geographical Information System (GIS) based spatial visualization resources in Kepler.

A scientific workflow combines data and processes into a configurable, structured set of steps that implement semi-automated computational solutions of a scientific data management or analysis problem. Scientific workflow systems provide graphical user interfaces to combine different technologies along with efficient methods for using them with the goal to increase the efficiency of the scientists. This tutorial provides an introduction to scientific workflow construction and management (Part I) and includes a detailed hands-on session (Part II) using the Kepler system. It is intended for an audience with a computational science background. It will cover principles and foundations of scientific workflows, Kepler environment installation, workflow construction using Kepler library components, and workflow execution management that uses Kepler facilities to provide process and data monitoring and provenance information, as well as high speed data movement solutions. This tutorial also incorporates hands-on exercises and application examples from different scientific disciplines.

Abstract Many scientific disciplines are now data and information driven, and new scientific knowledge is often gained by scientists putting together data analysis and knowledge discovery ‘pipelines’. A related trend is that more and more scientific communities realize the benefits of sharing their data and computational services, and are thus contributing to a distributed data and computational community infrastructure (a.k.a. ‘the Grid’). However, this infrastructure is only a means to an end and ideally scientists should not be too concerned with its existence. The goal is for scientists to focus on development and use of what we call scientific workflows . These are networks of analytical steps that may involve, e.g., database access and querying steps, data analysis and mining steps, and many other steps including computationally intensive jobs on high‐performance cluster computers. In this paper we describe characteristics of and requirements for scientific workflows as identified in a number of our application projects. We then elaborate on Kepler, a particular scientific workflow system, currently under development across a number of scientific data management projects. We describe some key features of Kepler and its underlying Ptolemy II system, planned extensions, and areas of future research. Kepler is a community‐driven, open source project, and we always welcome related projects and new contributors to join. Copyright © 2005 John Wiley & Sons, Ltd.

Existing grid workflow tools assume that individual components either communicate by passing files from one application to another, or are explicitly linked using interprocess communication pipes. When files are used it is usually difficult to overlap reader and write execution. On the other hand, interprocess communication primitives are invasive and require substantial modification to existing code. We have built a library, called GriddLeS, that combines the advantages of both approaches without any changes to the application code. GriddLeS overloads conventional file IO operations, and supports either local, remote or replicated file access, or direct communication over pipes. In this paper we discuss how GriddLeS can be combined with workflow packages and show how flexible and powerful virtual applications can be constructed rapidly. A large atmospheric science case study is discussed

Evolving technologies, as exemplified by computational grids and Web services, have made it possible to solve new scientific problems that would not have been feasible previously. In order to make such advances available to the community in general and to be able to solve new problems, not necessarily from the same discipline, it is imperative to build tools that provide a common user interface in order that application programmers and users do not have to be concerned with particulars of Web services and their underlying code, computational platforms, or with data file formats. We will describe our efforts in creating a computational chemistry environment that encompasses a general scientific workflow environment, a domain specific example for quantum chemistry, our ongoing design of a workflow user interface, and our efforts at database integration.

The article presents the Web services framework in the Kepler scientific workflow system and illustrates them with a real-world example.

Most scientists conduct analyses and run models in several different software and hardware environments, mentally coordinating the export and import of data from one environment to another. The Kepler scientific workflow system provides domain scientists with an easy-to-use yet powerful system for capturing scientific workflows (SWFs). SWFs are a formalization of the ad-hoc process that a scientist may go through to get from raw data to publishable results. Kepler attempts to streamline the workflow creation and execution process so that scientists can design, execute, monitor, re-run, and communicate analytical procedures repeatedly with minimal effort. Kepler is unique in that it seamlessly combines high-level workflow design with execution and runtime interaction, access to local and remote data, and local and remote service invocation. SWFs are superficially similar to business process workflows but have several challenges not present in the business workflow scenario. For example, they often operate on large, complex and heterogeneous data, can be computationally intensive and produce complex derived data products that may be archived for use in reparameterized runs or other workflows. Moreover, unlike business workflows, SWFs are often dataflow-oriented as witnessed by a number of recent academic systems (e.g., DiscoveryNet, Taverna and Triana) and commercial systems (Scitegic/Pipeline-Pilot, Inforsense). In a sense, SWFs are often closer to signal-processing and data streaming applications than they are to control-oriented business workflow applications.

We illustrate how a domain scientist can perform a complex scientific task by interleaving data access, querying, and manipulation, as well as analytical steps and computations in complex, problem specific ways. We show how our system is used by a geneticist for solving the problem of discovering the so-called "co-regulated" genes by interlinking data and computation from several Web sites, local computations, as well as local and remote databases. The main distinctive features of our system (compared, e.g., to the ZOO environment (Ioannidis et al., 1996)) include: (i) executable workflows run as Web services; (ii) abstract workflows employ concept names and semantic types that are higher-level (and thus more "scientist friendly") than executable workflows; and (iii) our system supports automatic translation of the latter into the former.

The authors present an approach for compiling "scientist-friendly" abstract workflow specifications into real-world executable workflows of Web service invocations, using a set of abstract-as-view definitions from a repository of abstract tasks. There have been a number of proposals and systems for scientific workflow management. However, our approach features unique aspects, in particular: the separation of abstract and concrete executable workflows; and the use of database mediation techniques to automatically translate AWFs into EWFs.