STRATUM is a Horizon Europe multi-partner project developing a clinically validated, real-time 3D decision support tool for brain tumour surgery. The system integrates Hyperspectral Imaging (HSI), AI-based multimodal data fusion, and heterogeneous High-Performance Computing (HPC) architectures combining Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and Processing-In-Memory (PIM) technologies. A touchless augmented reality interface facilitates safe and intuitive intraoperative interaction. The distinguishing characteristic of STRATUM is its end-to-end co-designed approach, which integrates advanced computing, state-of-the-art imaging and clinical expertise into a unified Point-of-Care (PoC) platform. Utilising a structured co-creation methodology involving surgeons, engineers, and social scientists, the project ensures usability, safety and regulatory compliance from its early design stages to its clinical validation. The usability of STRATUM will be tested in three hospitals located in different European regions with diverse conditions and regulations. This will allow to collect advice and remarks from surgical staff in a continuous co-creation and co-tuning protocol. Beyond its clinical objectives, STRATUM contributes to the advancement of heterogeneous computing for real-time diagnostics, AI acceleration in critical medical environments and energy-efficient system integration. Furthermore, it delivers open datasets, validated AI pipelines, and performance benchmarks with a view to fostering future research and industrial innovation in digital surgery. The STRATUM project establishes a replicable model for intelligent, human-centred computing integrating microelectronics, AI and medicine.The paper presents an overview of the project in terms of aims, concepts and technologies and the description of the state of the work when approaching the end of the second of the five years planned. Specifically, the outcomes of the steps related to the collaboration with surgeons and medical staff (co-creation process) and the intelligent Graphical User Interface (GUI) development will be described. The latter allows for contactless interaction of the surgeon with several functions that have already been developed in the system.

We introduce a structure-from-motion method specifically designed to process the raw micro-images captured by plenoptic 2.0 cameras. Unlike traditional monocular cameras, plenoptic cameras incorporate a micro-lens array between the sensor and the main lens, capturing depth information at the expense of a more complex set of parameters to evaluate. Instead of simply integrating their projection model into the classical structure-from-motion pipeline, our contribution identifies the pinhole cameras-driven constraints and takes advantage of the inherent disparity information present in plenoptic cameras. This facilitates a robust initialization of the reconstruction. Our method shortcuts two of the limitations of the classical structure-from-motion: the ambiguity found in scenes captured with low angular disparity and the scale ambiguity. It enables the reconstruction of scenes captured by multiple uncalibrated plenoptic cameras, without using any calibration pattern or subaperture view extraction step. Our method undergoes experimental validation on both natural and synthetic datasets, showing a 10% error accuracy for relative pose estimation, which is comparable to calibration-pattern based methods. The results are robust to coarse initialization. Contrary to classical structure-from-motion, it is able to reconstruct scenes with parallel facing cameras. It also shows greater accuracy than reconstruction methods based on pinhole camera conversion.

Immersive video is gaining relevance across various fields, but its integration into real applications remains limited due to the technical challenges of depth estimation. Generating accurate depth maps is essential for 3D rendering, yet high-quality algorithms can require hundreds of seconds to produce a single frame. While real-time depth estimation solutions exist — particularly monocular deep learning-based methods and active sensors such as time-of-flight or plenoptic cameras — their depth accuracy and multiview consistency are often insufficient for depth image-based rendering (DIBR) and immersive video applications. This highlights the persistent challenge of jointly achieving real-time performance and high-quality, correlated depth across views. This paper introduces eGoRG, a GPU-accelerated depth estimation algorithm based on MPEG DERS, which employs graph cuts to achieve high-quality results. eGoRG contributes a novel GPU-based graph cuts stage, integrating block-based push-relabel acceleration and a simplified alpha expansion method. These optimizations deliver quality comparable to leading graph-cut approaches while greatly improving speed. Evaluation on an MPEG multiview dataset and a static NeRF dataset demonstrates the algorithm’s effectiveness across different scenarios. • The proposal is a novel GPU-accelerated depth estimation algorithm based on graph cuts. • Algorithm-dependent strategies are introduced to maximize the quality–time trade-off. • Depth results are comparable to high-performing graph-cut approaches while being substantially faster. • The method is training-free and can process dynamic scenes. • The algorithm is a good trade-off between quality and processing time achieving near real-time results.

Neural Radiance Fields (NeRF) demonstrate impressive capabilities in rendering novel views of specific scenes by learning an implicit volumetric representation from posed RGB images without any depth information. View synthesis is the computational process of synthesizing novel images of a scene from different viewpoints, based on a set of existing images. One big problem is the need for a large number of images in the training datasets for neural network-based view synthesis frameworks. The challenge of data augmentation for view synthesis applications has not been addressed yet. NeRF models require comprehensive scene coverage in multiple views to accurately estimate radiance and density at any point. In cases without sufficient coverage of scenes with different viewing directions, cannot effectively interpolate or extrapolate unseen scene parts. In this paper, we introduce a new pipeline to tackle this data augmentation problem using depth data. We use MPEG's Depth Estimation Reference Software and Reference View Synthesizer to add novel non-existent views to the training sets needed for the NeRF framework. Experimental results show that our approach improves the quality of the rendered images using NeRF's model. The average quality increased by 6.4 dB in terms of Peak Signal-to-Noise Ratio (PSNR), with the highest increase being 11 dB. Our approach not only adds the ability to handle the sparsely captured multiview content to be used in the NeRF framework, but also makes NeRF more accurate and useful for creating high-quality virtual views.

Integrated digital diagnostics can support complex surgical procedures in many anatomical sites, brain tumour surgery being the most complex. STRATUM is a 5-year Horizon Europe funded project with the goal of developing an innovative 3D decision support tool for brain tumour surgeries, based on real-time multimodal data processing using artificial intelligence algorithms. The proposed tool is envisioned as an energy-efficient Point-of-Care computing system to be integrated within neurosurgical workflows to aid surgeons to make informed, efficient, and accurate decisions during surgical procedures. The expected long-term impact of STRATUM is to reduce the duration of surgical procedures, thus decreasing patients' risks, but also optimising the resources of European health care systems.

Neural Radiance Fields (NeRF) have demonstrated exceptional performance in generating novel views of scenes by learning implicit volumetric representations from calibrated RGB images, without depth information. A major limitation is the need for large training datasets in neural network-based view synthesis frameworks. The challenge of effective data augmentation for view synthesis remains unresolved. NeRF models require extensive scene coverage from multiple views to accurately estimate radiance and density. Insufficient coverage reduces the model’s ability to interpolate or extrapolate unseen parts of the scene effectively. In this paper, we propose a novel pipeline to address this data augmentation issue using depth map information. We use depth image-based rendering (DIBR) to overcome the lack of enough views for training NeRF. Experimental results indicate that our approach enhances the quality of rendered images using the NeRF framework, achieving an average peak signal-to-noise ratio (PSNR) increase of 7.2 dB, with a maximum improvement of 12 dB.

Hyperspectral imaging analyzed by machine learning algorithms is a powerful tool to classify materials, tissues, molecules and pathogens. By analyzing the electromagnetic spectrum of liquid serum samples, it has been demonstrated that it is possible to predict which patients with possible head trauma injury will have a possible result on computer tomography. This process is being carried out with very complex, slow and expensive spectrometric techniques. To tackle this problem, this study presents a simple hyperspectral imaging system that allows the capture of multiple serum samples with one single scan, without light artifacts as it works in transmittance and without a high data redundancy rate. Throughout this paper, the main characteristics of this system, the preprocessing chain necessary to extract the information from these captures, the working methodology, and the analysis performed are presented. Hyperspectral images of plasma from 405 patients were captured and the signatures obtained from this system were compared with the signatures captured by a spectrometer, which served as a reference system. With a mean correlation of 97.3% and a standard deviation of 3.6%, the presented system not only captures correctly liquid samples, but also provides spatial information and can capture many more samples in a single scan. In addition, a statistical study is presented on which spectrum bands present a higher concentration of information, which will be very beneficial for future analysis.

In the field of hyperspectral imaging, accurate and reliable data analysis is essential for many applications, including medicine, remote sensing, and material science. White calibration is a critical step in this process, as it accounts for deviations in the light source intensity and spectral distribution. Nevertheless, it is important to note that the specific geometry of the scene plays a crucial role in the white calibration, affecting the angle of incidence of the light and, subsequently, the measured spectral response. By taking surface normals and depth information into account during white calibration, we can ensure that subsequent hyperspectral images are accurately calibrated and comparable, leading to more robust and meaningful data analysis. For that matter, in this paper, we demonstrate a methodology to fuse hyperspectral and depth information, and how this fusion can help to correct different geometrical properties of the analyzed sample. A series of laboratory experiments were conducted on samples with different geometries and surface properties. Specifically, the error dispersion of the spectral signatures was reduced from 12 % to less than 4 %, a substantial improvement that highlights the potential of the proposed method.

Over the last two decades, augmented reality (AR) has led to the rapid development of new interfaces in various fields of social and technological application domains. One such domain is medicine, and to a higher extent surgery, where these visualization techniques help to improve the effectiveness of preoperative and intraoperative procedures. Following this trend, this paper presents SLIMBRAIN, a real-time acquisition and processing AR system suitable to classify and display brain tumor tissue from hyperspectral (HS) information. This system captures and processes HS images at 14 frames per second (FPS) during the course of a tumor resection operation to detect and delimit cancer tissue at the same time the neurosurgeon operates. The result is represented in an AR visualization where the classification results are overlapped with the RGB point cloud captured by a LiDAR camera. This representation allows natural navigation of the scene at the same time it is captured and processed, improving the visualization and hence effectiveness of the HS technology to delimit tumors. The whole system has been verified in real brain tumor resection operations.

Hyperspectral (HS) imaging (HSI) techniques have demonstrated to be useful in the medical field to characterize tissues without any contact and without ionizing the patient. Besides, HSI combined with supervised machine learning (ML) algorithms have proven to be an effective technique to assist neurosurgeons to resect brain tumors. This research looks at the effects of hyperparameter optimization on two common supervised ML algorithms used for brain tumor classification: support vector machines (SVM) and random forest (RF). Correctly classifying brain tumor with HS data containing low spatial and spectral information can be challenging. To tackle this problem, this study has applied hyperparameter optimization techniques on SVM and RF with 10 brain images of patients suffering from glioblastoma multiforme (GBM) with non-mutated isocitrate dehydrogenase (IDH) enzymes. These captures have 409x217 spatial resolution and 25 normalized reflectance wavelengths gathered from 665 to 960 nm with a HS snapshot camera. Results show how this work has been able to obtain 98,60% of weighted area under the curve (AUC) on the test score by employing naive optimizations like grid search (GS) or random search (RS) and even more complex methods based on Bayesian optimization (BO). Not only the weighted AUC of SVM has been improved by 8%, but BO have also enhanced the AUC of the tumor class by 22.50% in comparison with non-optimized SVM models in the state-of-the-art, achieving AUC values of 95,49% on the tumor class. Furthermore, these improvements have been illustrated with classification maps to demonstrate the importance of hyperparameter optimization on SVM to clearly classify brain tumor, whereas non-optimized models from previous studies are unable to detect the tumor.

In this paper, we propose a runtime framework that implements code multi-versioning and specialization to optimize and parallelize loop kernels that are invoked many times with varying parameters. These parameters may influence the code structure, the touched memory locations, the workload, and the runtime performance. They may also impact the validity of the parallelizing and optimizing polyhedral transformations that are applied on-the-fly.

Cyber-Physical Systems (CPS) are embedded computational collaborating devices, capable of sensing and controlling physical elements and, often, responding to humans. Designing and managing systems able to respond to different, concurrent requirements during operation is not straightforward, and introduce the need of proper support at design-time and run-time. The Cross-layer modEl-based fRamework for multi-oBjective dEsign of Reconfigurable systems in unceRtain hybRid envirOnments (CERBERO) EU project has developed a design environment for adaptive CPS. CERBERO approach leverages on model-based methodologies including different technologies and tools developed to cover design and operation from user interactions down to low level computing layer implementation.

Stream processing applications running on Heterogeneous Multi-Processor Systems on Chips (HMPSoCs) require efficient resource allocation and management, both at compile-time and at runtime. To cope with modern adaptive applications whose behavior can not be exhaustively predicted at compile-time, runtime managers must be able to take resource allocation decisions on-the-fly, with a minimum overhead on application performance. Resource allocation algorithms often rely on an internal modeling of an application. Directed Acyclic Graph (DAGs) are the most commonly used models for capturing control and data dependencies between tasks. DAGs are notably often used as an intermediate representation for deploying applications modeled with a dataflow Model of Computation (MoC) on HMPSoCs. Building such intermediate representation at runtime for massively parallel applications is costly both in terms of computation and memory overhead. In this paper, an intermediate representation of DAGs for resource allocation is presented. This new representation shows improved performance for run-time analysis of dataflow graphs with less overhead in both computation time and memory footprint. The performances of the proposed representation are evaluated on a set of computer vision and machine learning applications.

The widening of the complexity-productivity gap witnessed in the last years is becoming unaffordable from the application development point of view. New design methods try to automate most designers tasks in order to bridge this gap. In addition, new Models of Computation (MoC), as those dataflow-based, ease the expression of parallelism within applications and lead to higher productivity.

This paper describes an OMAP based real-time test bench to find the Pareto frontier of an H.264/SVC decoder within a distortion-energy optimization space. A metric to estimate video distortion is introduced. In addition, energy consumption estimates are obtained from real-time measurements of the computational load. Finally, test bench operation is successfully demonstrated with different H.264/SVC-compliant sets of sequences.

In this paper, an architecture for multimedia stream processing and communication is presented. The system is aimed to communicate MPEG4 multimedia applications over generic networks. Main features of the architecture are scalability in the number of multimedia streams managed, bandwidth sharing, capacity to control the offered service quality and possibility to implement mobile applications. A chip area network (CIAN) is used to interconnect the different architectural elements. Four main units are distinguished in the design: cell communication, QoS control, protocol processing and DMA (Direct Memory Access). Preliminary results of an implementation of the cell communication unit as an ATM-cell-based multiplexing one show the suitability of the architecture for STS-12/STM-4/OC-12 throughputs (622.08 Mb/s).

The implementation of a FPGA based coprocessor and its programming methodology are shown. The effects of different sequencing models, and regular and irregular circuits on the hardware and in the programming methodology are discussed. Two examples are described: a sorting network and the kernel of a speech recognition algorithm. The results are still preliminary but they suggest some architectural improvements for general FPGA based computing machines.