We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO). Unlike other multimodal models, TerraMind is pretrained on dual-scale representations combining both token-level and pixel-level data across modalities. On a token level, TerraMind encodes high-level contextual information to learn cross-modal relationships, while on a pixel level, TerraMind leverages fine-grained representations to capture critical spatial nuances. We pretrained TerraMind on nine geospatial modalities of a global, large-scale dataset. In this paper, we demonstrate that (i) TerraMind's dual-scale early fusion approach unlocks a range of zero-shot and few-shot applications for Earth observation, (ii) TerraMind introduces "Thinking-in-Modalities" (TiM) -- the capability of generating additional artificial data during finetuning and inference to improve the model output -- and (iii) TerraMind achieves beyond state-of-the-art performance in community-standard benchmarks for EO like PANGAEA. The pretraining dataset, the model weights, and our code are open-sourced under a permissive license.

Image retrieval enables an efficient search through vast amounts of satellite imagery and returns similar images to a query. Deep learning models can identify images across various semantic concepts without the need for annotations. This work proposes to use Geospatial Foundation Models, like Prithvi, for remote sensing image retrieval with multiple benefits: i) the models encode multi-spectral satellite data and ii) generalize without further fine-tuning. We introduce two datasets to the retrieval task and observe a strong performance: Prithvi processes six bands and achieves a mean Average Precision of 97.62% on BigEarthNet-43 and 44.51% on ForestNet-12, outperforming other RGB-based models. Further, we evaluate three compression methods with binarized embeddings balancing retrieval speed and accuracy. They match the retrieval speed of much shorter hash codes while maintaining the same accuracy as floating-point embeddings but with a 32-fold compression. The code is available at https://github.com/IBM/remote-sensing-image-retrieval.

As repositories of large scale data in earth observation (EO) have grown, so have transfer and storage costs for model training and inference, expending significant resources. We introduce Neural Embedding Compression (NEC), based on the transfer of compressed embeddings to data consumers instead of raw EO data. We adapt foundation models (FM) through learned neural compression to generate multi-task embeddings while navigating the tradeoff between compression rate and embedding utility. We update only a small fraction of the FM parameters ( 10%) for a short training period (∼1% of the iterations of∼pre-training). We evaluate NEC on two EO tasks: scene classification and semantic segmentation. Compared with applying traditional compression to the raw data, NEC achieves similar accuracy with a 75% to 90% reduction in data. Even at 99.7% compression, performance drops by only 5% on the scene classification task. Overall, NEC is a data-efficient yet performant approach for multi-task EO modelling.

Climate change results in an increased probability of extreme weather events that put societies and businesses at risk on a global scale. Therefore, near real-time mapping of natural hazards is an emerging priority for the support of natural disaster relief, risk management, and informed governmental policy decisions. Current remote sensing based approaches to near real-time natural hazard mapping increasingly leverage advantage of deep learning (DL). Nevertheless, DL-based approaches are mainly designed for one specific task in a single geographic region based on specific frequency bands of satellite data. For that reason, DL models used to map specific natural hazards struggle with their generalization to other types of natural hazards in unseen regions. In this work, we propose a methodology to significantly improve the generalizability of DL natural hazards mappers based on pre-training on a suitable pre-task. Without access to any data from the target domain, we demonstrate that this methodology improved generalizability across four U-Net architectures for the segmentation of unseen natural hazards, such as flood events, landslides, and massive glacier collapses. Importantly, our method is strongly invariant to geographic differences and the type of input frequency bands of satellite data. That is confirmed by obtaining a balanced accuracy of up to 0.74 in comparison with performance of reference baselines. By leveraging characteristics of unlabeled images from the target domain that are publicly available, our approach is able to further improve the generalization behavior of DL models without fine-tuning. That is reflected in performance metrics. Thereby, our approach is one of first attempts to support the development of foundation models for earth monitoring with the objective of directly segmenting unseen natural hazards across novel geographic regions from different sources of satellite imagery.

Floods are among the most costly weather hazards for societies and businesses globally. With increasing global warming, these events have become even more frequent and more devastating. Thus, accurate flood mapping has become critical for disaster relief, risk management and mitigation. Current flood segmentation methods use either threshold-based approaches or deep-learning schemes, e.g. using the U-Net architecture, to differentiate between water-covered bodies or dry land on Earth observation images. Many schemes are exploiting imagery from synthetic aperture radar (e.g. Sentinel 1 satellites) or visual bands of satellites such as the Sentinel 2, but often restrict themselves to using one or very few modalities, i.e. spectral wavelengths, despite the availability of many more wavelengths or pre-processed indices with potential value to the challenge. In support of operationalizing flood segmentation on a global scale using deep learning, we propose semantic flood segmentation exploiting optionally many different modalities (i.e. multimodal flood segmentation), making the approach largely immune to geographic differences across the globe. Using U-Net at the core of our work, we observe very good generalisation of our segmentation model to unseen flood events in our holdout set at the level of 0.95 F1 Score (0.92 IoU) for both no water and water class, and 0.53 F1 Score (0.43 IoU) for water class, respectively.

Surface water mapping in satellite images enables flood monitoring, a task with increasing importance under changing climate conditions. Current segmentation methods based on Deep Learning require large, curated datasets for training’ which are difficult to obtain. In this paper, we present a probabilistic approach to water segmentation based on established computer vision methods that requires little training and is easy to interpret. We use prior knowledge of typical backscatter intensity to locate seed pixels likely to be in water and land. A preliminary rough segmentation using thresholding guides the selection of two image patches that will be fully labeled using seeded region growing segmentation. Then, we sample small patches within the automatically labeled regions to train a fully connected neural network that, running in sliding windows, scores for the presence of water in the entire image. The approach is tested for mapping surface water during a large flood event in Aude, France. Outputs of the proposed approach are compared to the reference flood delineation map provided online by the Copernicus service. A F1 score of 0.67 suggests that performance of the proposed approach is similar or better than classic thresholding methods used as benchmark.

In this paper, we introduce a home-based COPD management system to objectively track the disease progress during the daily-living of patients. The selection of the devices is performed to map the COPD assessment test questions and the patients lung function at home. A CAir Desk was developed to improve the usability of the many connected devices. We explain the IT architecture, data flow, as well as the data aggregation and visualization concept of our solution. Details on the usage during the trial is provided as well.

In this paper, we report on a mobile-health intervention study for COPD patients. Mobile devices are used to continuously record relevant patient symptoms, such as lung function, cough intensity, sputum color, vital-signs and activity. Furthermore, behavioral support is provided by a bi-directional communication channel and virtual assistant communication. The goal is to train a predictive algorithm to prevent exacerbations and to provide personalized virtual coaching to improve medication adherence and activity level.

Wearables that continuously acquire vital and other medically relevant parameters facilitate treatment optimizations for individual patients and reduce the duration of hospitalizations - thus improving the patients' quality of life. To accomplish this, we demonstrate a scalable architecture that connects wearables through a hub to the cloud, combines edge and cloud computing to provide optimal user interaction, and allows analytics on multi-stream data from those connected devices.

Liquid-cooling using microchannel heat sinks etched on silicon dies is seen as a promising solution to the rising heat fluxes in two-dimensional and stacked three-dimensional integrated circuits. Development of such devices requires accurate and fast thermal simulators suitable for early-stage design. To this end, we present 3D-ICE, a compact transient thermal model (CTTM), for liquid-cooled ICs. 3D-ICE was first advanced incorporating the 4-resistor model-based CTTM (4RM-based CTTM). Later, it was enhanced to speed up simulations and to include complex heat sink geometries such as pin fins using the new 2 resistor model (2RM-based CTTM). In this paper, we extend the 3D-ICE model to include liquid-cooled ICs with multi-port cavities, i.e., cavities with more than one inlet and one outlet ports, and non-straight microchannels. Simulation studies using a realistic 3D multiprocessor system-on-chip (MPSoC) with a 4-port microchannel cavity highlight the impact of using 4-port cavity on temperature and also demonstrate the superior performance of 2RM-based CTTM compared to 4RM-based CTTM. We also present an extensive review of existing literature and the derivation of the 3D-ICE model, creating a comprehensive study of liquid-cooled ICs and their thermal simulation from the perspective of computer systems design. Finally, the accuracy of 3D-ICE has been evaluated against measurements from a real liquid-cooled 3D-IC, which is the first such validation of a simulator of this genre. Results show strong agreement (average error${\bf \lt 10\%}$), demonstrating that 3D-ICE is an effective tool for early-stage thermal-aware design of liquid-cooled 2D-/3D-ICs.

Chip microscale liquid-cooling reduces thermal resistance and improves datacenter efficiency with higher coolant temperatures by eliminating chillers and allowing thermal energy re-use in cold climates. Liquid cooling enables an unprecedented density in future computers to a level similar to a human brain. This is mediated by a dense 3D architecture for interconnects, fluid cooling, and power delivery of energetic chemical compounds transported in the same fluid. Vertical integration improves memory proximity and electrochemical power delivery creating valuable space for communication. This strongly improves large system efficiency thereby allowing computers to grow beyond exa-scale.

Two-phase liquid cooling of computer chips via microchannels etched directly on silicon dies is a potential long-term solution to enable continued integration of high-performance multiprocessors. Two-phase cooling refers to the heat removal via evaporation of a refrigerant flowing inside a heat sink. While possessing superior cooling properties, large-scale use of this technology in the industry is limited by the lack of thermal modeling tools that can accurately predict temperatures in a two-phase cooled IC. In this paper, we propose STEAM, a new compact thermal model for 2D/3D ICs with two-phase cooling via silicon microchannels. The accuracy of the STEAM model is validated against measurements from a real two-phase cooled IC test stack reported previously in literature. Temperatures were predicted with an average error as low as 10.2% for uniform heat fluxes and 6.9% for hotspots. Finally, the STEAM model is applied to a realistic 3D multiprocessor system-on-chip (3D MP-SoC) with two-phase cooling to simulate IC temperatures and the refrigerant pumping power, demonstrating the applicability of STEAM in the early-stage design of near-future high-performance computers with two-phase cooling.

New tendencies envisage 3D Multi-Processor System-On-Chip (MPSoC) design as a promising solution to keep increasing the performance of the next-generation high-performance computing (HPC) systems. However, as the power density of HPC systems increases with the arrival of 3D MPSoCs, supplying electrical power to the computing equipment and constantly removing the generated heat is rapidly becoming the dominant cost in any HPC facility. Thus, both power and thermal/cooling implications play a major role in the design of new HPC systems, given the energy constraints in our society. Therefore, EPFL, IBM and ETHZ have been working within the CMOSAIC Nano-Tera.ch program project in the last three years on the development of a holistic thermally-aware design. This paper presents the exploration in CMOSAIC of novel cooling technologies, as well as suitable thermal modeling and system-level design methods, which are all necessary to develop 3D MPSoCs with inter-tier liquid cooling systems. As a result, we develop energy-efficient run-time thermal control strategies to achieve energy-efficient cooling mechanisms to compress almost 1 Tera nano-sized functional units into one cubic centimeter with a 10 to 100 fold higher connectivity than otherwise possible. The proposed thermally-aware design paradigm includes exploring the synergies of hardware-, software- and mechanical-based thermal control techniques as a fundamental step to design 3D MPSoCs for HPC systems. More precisely, we target the use of inter-tier coolants ranging from liquid water and two-phase refrigerants to novel engineered environmentally friendly nano-fluids, as well as using specifically designed micro-channel arrangements, in combination with the use of dynamic thermal management at system-level to tune the flow rate of the coolant in each micro-channel to achieve thermally-balanced 3D-ICs. Our management strategy prevents the system from surpassing the given threshold temperature while achieving up to 67% reduction in cooling energy and up to 30% reduction in system-level energy in comparison to setting the flow rate at the maximum value to handle the worst-case temperature.

3-D stacked systems reduce communication delay in multiprocessor system-on-chips (MPSoCs) and enable heterogeneous integration of cores, memories, sensors, and RF devices. However, vertical integration of layers exacerbates temperature-induced problems such as reliability degradation. Liquid cooling is a highly efficient solution to overcome the accelerated thermal problems in 3-D architectures; however, it brings new challenges in modeling and run-time management for such 3-D MPSoCs with multitier liquid cooling. This paper proposes a novel design-time/run-time thermal management strategy. The design-time phase involves a rigorous thermal impact analysis of various thermal control variables. We then utilize this analysis to design a run-time fuzzy controller for improving energy efficiency in 3-D MPSoCs through liquid cooling management and dynamic voltage and frequency scaling (DVFS). The fuzzy controller adjusts the liquid flow rate dynamically to match the cooling demand of the chip for preventing overcooling and for maintaining a stable thermal profile. The DVFS decisions increase chip-level energy savings and help balance the temperature across the system. Our controller is used in conjunction with temperature-aware load balancing and dynamic power management strategies. Experimental results on 2-tier and 4-tier 3-D MPSoCs show that our strategy prevents the system from exceeding the given threshold temperature. At the same time, we reduce cooling energy by up to 63% and system-level energy by up to 21% in comparison to statically setting a flow rate setting to handle worst-case temperatures.

Liquid cooling has emerged as a promising solution for addressing the elevated temperatures in 3D stacked architectures. In this work, we first propose a framework for detailed thermal modeling of the microchannels embedded between the tiers of the 3D system. In multicore systems, workload varies at runtime, and the system is generally not fully utilized. Thus, it is not energy-efficient to adjust the coolant flow rate based on the worst-case conditions, as this would cause an excess in pump power. For energy-efficient cooling, we propose a novel controller to adjust the liquid flow rate to meet the desired temperature and to minimize pump energy consumption. Our technique also includes a job scheduler, which balances the temperature across the system to maximize cooling efficiency and to improve reliability. Our method guarantees operating below the target temperature while reducing the cooling energy by up to 30%, and the overall energy by up to 12% in comparison to using the highest coolant flow rate.

Three dimensional stacked integrated circuits (3D ICs) are extremely attractive for overcoming the barriers in interconnect scaling, offering an opportunity to continue the CMOS performance trends for the next decade. However, from a thermal perspective, vertical integration of high-performance ICs in the form of 3D stacks is highly demanding since the effective areal heat dissipation increases with number of dies (with hotspot heat fluxes up to 250 W/cm2) generating high chip temperatures. In this context, inter-tier integrated microchannel cooling is a promising and scalable solution for high heat flux removal. A robust design of a 3D IC and its subsequent thermal management depend heavily upon accurate modeling of the effects of liquid cooling on the thermal behavior of the IC during the early stages of design. In this paper we present 3D-ICE, a compact transient thermal model (CTTM) for the thermal simulation of 3D ICs with multiple inter-tier microchannel liquid cooling. The proposed model is compatible with existing thermal CAD tools for ICs, and offers significant speed-up (up to 975x) over a typical commercial computational fluid dynamics simulation tool while preserving accuracy (i.e., maximum temperature error of 3.4%). In addition, a thermal simulator has been built based on 3D-ICE, which is capable of running in parallel on multicore architectures, offering further savings in simulation time and demonstrating efficient parallelization of the proposed approach.