Volume electron microscopy (EM) now enables nanometric-scale 3D reconstructions of neural tissue, opening the door to quantitative, morphology-driven neuroscience beyond connectivity alone. While previous studies relied on handcrafted descriptors and classical machine learning for morphology analysis, recent progress in deep learning for 3D shape understanding offers new opportunities to learn robust, task-specific representations directly from geometric data. In this paper we present NeMoCo , a geometry learning framework that targets the key practical bottleneck in connectomics and ultrastructural analysis: the scarcity and cost of dense expert annotations for the long tail of neurite and organelle phenotypes. NeMoCo formulates representation learning for EM-derived neurite meshes in a self-supervised Momentum Contrast (MoCo) style. We use DiffusionNet (Sharp et al., 2022) as a mesh encoder with intrinsic spectral descriptors (HKS) and train with a momentum-updated teacher encoder and a large memory bank of negatives. To learn invariances that are essential in practice, we generate paired geometric views via controlled affine transformations and resolution changes (including mesh decimation), encouraging embeddings to be stable under nuisance variability while remaining discriminative. We provide an extensive study of augmentation strength and temperature, and evaluate learned representations through frozen retrieval and non-parametric classification (frozen kNN), as well as downstream supervised fine-tuning under limited labels. NeMoCo demonstrates that MoCo-style self-supervision yields robust neurite morphology embeddings on EM meshes, improving label-efficiency and offering a scalable foundation for retrieval, clustering, and phenotype discovery in ultrastructural neuroscience. All the data and the code used for NeMoCo are available at https://github.com/Uzshah/NeMoCo .

Abstract Food computing has emerged as a promising research field, employing artificial intelligence, deep learning, and data science methodologies to enhance various stages of food production pipelines. To this end, the food computing community has compiled a variety of data sets and developed various deep-learning architectures to perform automatic classification. However, automated food classification presents a significant challenge, particularly when it comes to local and regional cuisines, which are often underrepresented in available public-domain data sets. Nevertheless, obtaining high-quality, well-labeled, and well-balanced real-world labeled images is challenging since manual data curation requires significant human effort and is time-consuming. In contrast, the web has a potentially unlimited source of food data but tapping into this resource has a good chance of corrupted and wrongly labeled images. In addition, the uneven distribution among food categories may lead to data imbalance problems. All these issues make it challenging to create clean data sets for food from web data. To address this issue, we present AutoCleanDeepFood, a novel end-to-end food computing framework for regional gastronomy that contains the following components: (i) a fully automated pre-processing pipeline for custom data sets creation related to specific regional gastronomy, (ii) a transfer learning-based training paradigm to filter out noisy labels through loss ranking, incorporating a Russian Roulette probabilistic approach to mitigate data imbalance problems, and (iii) a method for deploying the resulting model on smartphones for real-time inferences. We assess the performance of our framework on a real-world noisy public domain data set, ETH Food-101, and two novel web-collected datasets, MENA-150 and Pizza-Styles. We demonstrate the filtering capabilities of our proposed method through embedding visualization of the feature space using the t-SNE dimension reduction scheme. Our filtering scheme is efficient and effectively improves accuracy in all cases, boosting performance by 0.96, 0.71, and 1.29% on MENA-150, ETH Food-101, and Pizza-Styles, respectively.

Abstract Food plays an important role in our lives that goes beyond mere sustenance. Food affects behavior, mood, and social life. It has recently become an important focus of multimedia and social media applications. The rapid increase of available image data and the fast evolution of artificial intelligence, paired with a raised awareness of people’s nutritional habits, have recently led to an emerging field attracting significant attention, called food computing, aimed at performing automatic food analysis. Food computing benefits from technologies based on modern machine learning techniques, including deep learning, deep convolutional neural networks, and transfer learning. These technologies are broadly used to address emerging problems and challenges in food-related topics, such as food recognition, classification, detection, estimation of calories and food quality, dietary assessment, food recommendation, etc. However, the specific characteristics of food image data, like visual heterogeneity, make the food classification task particularly challenging. To give an overview of the state of the art in the field, we surveyed the most recent machine learning and deep learning technologies used for food classification with a particular focus on data aspects. We collected and reviewed more than 100 papers related to the usage of machine learning and deep learning for food computing tasks. We analyze their performance on publicly available state-of-art food data sets and their potential for usage in multimedia food-related applications for various needs (communication, leisure, tourism, blogging, reverse engineering, etc.). In this paper, we perform an extensive review and categorization of available data sets: to this end, we developed and released an open web resource in which the most recent existing food data sets are collected and mapped to the corresponding geographical regions. Although artificial intelligence methods can be considered mature enough to be used in basic food classification tasks, our analysis of the state-of-the-art reveals that challenges related to the application of this technology need to be addressed. These challenges include, among others: poor representation of regional gastronomy, incorporation of adaptive learning schemes, and reverse engineering for automatic food creation and replication.

Pump and Dump schemes represent a threat to any market. While this issue has long been regulated in mature markets, in unregulated markets, such as crypto exchanges, this plague is very present, and even exacerbated by the low capitalizaton of many cryptocurrencies that represent the perfect target for such a fraudulent scheme. In this paper, we detail a Low Latency Detection solution (LLD) based on deep learning to automatically detect pump and dump activities on centralized cryptocurrency exchanges. We train a LSTM-based auto-encoder on BTC valuations, which can reliably be considered a proxy for regular trading-due to their larger capitalization. We use this auto-encoder to predict valuations on alt coins and use thresholding on a Gaussian tail condition to trigger detection. We argue that low latency detection is paramount for the practicality of such approaches. Unlike previous methods, our solution (LLD) detects the majority of pumps in less than five minutes (2.2 minutes on average) when using OHLCV data at one-minute resolution. In addition, we use social media data only to generate ground truths during testing. We show that in many cases a significant amount of the trade volume could have been saved had LLD been used to trigger trade suspension mechanisms. The idiosyncratic approach of our scheme, its sound rationale and viability, combined with the quality of achieved results-tested over an extensive experimental campaign-and the insights discussed in the paper also pave the way for further research in the field.

We present a novel automatic few-shot selection (FSS) approach with a noisy web-collected dataset for underwater tropical fishes. Underwater image segmentation is of utmost importance in marine biology and underwater robotics. However, obtaining high-quality annotated underwater images is a challenging task due to the scarcity of underwater imaging systems and the high cost of data acquisition. Alternatively, web data is a readily available source of images, but it frequently contains web-corrupted noisy labels, making it challenging to curate a clean dataset for underwater tropical fish segmentation. Generally, the manual selection of high-quality images from web data requires human supervision and expert proofreading, which is both expensive and time-consuming. To address this issue, we propose Noise2Seg, an automatic processing framework composed by: (i) an automated web scrapping tool, (ii) a robust and automatic FSS process using a loss ranking scheme; (iii) a manual annotation component using “Roboflow”, (iv) the latest You Only Look Once version 8 (YOLOv8) for underwater fish detection and segmentation. Additionally, we curated and annotated a novel dataset for the segmentation of tropical fishes from Qatar marine ecosystem (Qatar Tropical Fishes-10 QTF-10). We compared the performance of manual and automatic FSS using mean average precision (mAP) score; manual FSS achieved all mAP (91.5%, 93.4%), and minimum mAP (19.5%, 54.8%), while automatic FSS achieved all mAP (92%, 99.5%), and minimum mAP (24.9%, 99.5%) with 5 and 10 images per class, respectively. The code and dataset related to this paper can be found on GitHub11https://github.com/GilalNauman/Automatic-Few-shot-Selction-ISNCC/tree/main, accessed on 1st of March 2023.

Today’s Extended Reality (XR) applications that call for specific Diminished Reality (DR) strategies to hide specific classes of objects are increasingly using 360° cameras, which can capture entire areas in a single picture. In this work, we present an interactive-based image processing, editing and rendering system named SPIDER, that takes a spherical 360° indoor scene as input. The system is composed of a novel integrated deep learning architecture for extracting geometric and semantic information of full and empty rooms, based on gated and dilated convolutions, followed by a super-resolution module for improving the resolution of the color and depth signals. The obtained high resolution representations allow users to perform interactive exploration and basic editing operations on the reconstructed indoor scene, namely: (i) rendering of the scene in various modalities (point cloud, polygonal, wireframe) (ii) refurnishing (transferring portions of rooms) (iii) deferred shading through the usage of precomputed normal maps. These kinds of scene editing and manipulations can be used for assessing the inference from deep learning models and enable several Mixed Reality applications in areas such as furniture retails, interior designs, and real estates. Moreover, it can also be useful in data augmentation, arts, designs, and paintings. We report on the performance improvement of the various processing components on public domain spherical image indoor datasets.

We introduce a novel deep neural network to estimate a depth map from a single monocular indoor panorama. The network directly works on the equirectangular projection, exploiting the properties of indoor 360° images. Starting from the fact that gravity plays an important role in the design and construction of man-made indoor scenes, we propose a compact representation of the scene into vertical slices of the sphere, and we exploit long- and short-term relationships among slices to recover the equirectangular depth map. Our design makes it possible to maintain high-resolution information in the extracted features even with a deep network. The experimental results demonstrate that our method outperforms current state-of-the-art solutions in prediction accuracy, particularly for real-world data.

We present a shape processing framework for visual exploration of cellular nuclear envelopes extracted from microscopic images arising in histology and neuroscience. The framework is based on a novel shape descriptor of closed contours in 2D and 3D. In 2D, it relies on a geodesically uniform resampling of discrete curves to compute unsigned curvatures at vertices and edges based on discrete differential geometry. Our descriptor is, by design, invariant under translation, rotation, and parameterization. We achieve the latter invariance under parameterization shifts by using elliptic Fourier analysis on the resulting curvature vectors. Uniform scale-invariance is optional and is a result of scaling curvature features to z-scores. We further augment the proposed descriptor with feature coefficients obtained through sparse coding of the extracted cellular structures using K-sparse autoencoders. For the analysis of 3D shapes, we compute mean curvatures based on the Laplace-Beltrami operator on triangular meshes, followed by computing a spherical parameterization through mean curvature flow. Finally, we compute the Spherical Harmonics decomposition to obtain invariant energy coefficients. Our invariant descriptors provide an embedding into a fixed-dimensional feature space that can be used for various applications, e.g., as input features for deep and shallow learning techniques or as input for dimension reduction schemes to provide a visual reference for clustering shape collections. We demonstrate the capabilities of our framework in the context of visual analysis and unsupervised classification of 2D histology images and 3D nuclear envelopes extracted from serial section electron microscopy stacks.

Internet of Things (IoT) is an innovative paradigm envisioned to provide massive applications that are now part of our daily lives. Millions of smart devices are deployed within complex networks to provide vibrant functionalities, including communications, monitoring, and controlling of critical infrastructures. However, this massive growth of IoT devices and the corresponding huge data traffic generated at the edge of the network created additional burdens on the state-of-the-art centralized cloud computing paradigm due to the bandwidth and resource scarcity. Hence, edge computing (EC) is emerging as an innovative strategy that brings data processing and storage near to the end users, leading to what is called the EC-assisted IoT. Although this paradigm provides unique features and enhanced Quality of Service (QoS), it also introduces huge risks in data security and privacy aspects. This article conducts a comprehensive survey on security and privacy issues in the context of EC-assisted IoT. In particular, we first present an overview of EC-assisted IoT, including definitions, applications, architecture, advantages, and challenges. Second, we define security and privacy in the context of EC-assisted IoT. Then, we extensively discuss the major classifications of attacks in EC-assisted IoT and provide possible solutions and countermeasures along with the related research efforts. After that, we further classify some security and privacy issues as discussed in the literature based on security services and based on security objectives and functions. Finally, several open challenges and future research directions for secure EC-assisted IoT paradigm are also extensively provided.

In this paper, we present a novel data structure, called the Mixture Graph. This data structure allows us to compress, render, and query segmentation histograms. Such histograms arise when building a mipmap of a volume containing segmentation IDs. Each voxel in the histogram mipmap contains a convex combination (mixture) of segmentation IDs. Each mixture represents the distribution of IDs in the respective voxel's children. Our method factorizes these mixtures into a series of linear interpolations between exactly two segmentation IDs. The result is represented as a directed acyclic graph (DAG) whose nodes are topologically ordered. Pruning replicate nodes in the tree followed by compression allows us to store the resulting data structure efficiently. During rendering, transfer functions are propagated from sources (leafs) through the DAG to allow for efficient, pre-filtered rendering at interactive frame rates. Assembly of histogram contributions across the footprint of a given volume allows us to efficiently query partial histograms, achieving up to 178 x speed-up over naive parallelized range queries. Additionally, we apply the Mixture Graph to compute correctly pre-filtered volume lighting and to interactively explore segments based on shape, geometry, and orientation using multi-dimensional transfer functions.

We present a novel and efficient approach for computing joint graph layouts and then use it to visualize collections of segmented meshes. Our joint graph layout algorithm takes as input the adjacency matrices for a set of graphs along with partial, possibly soft, correspondences between nodes of different graphs. We then use a two stage procedure, where in the first step, we extend spectral graph drawing to include a consistency term so that a collection of graphs can be handled jointly. Our second step extends metric multi-dimensional scaling with stress majorization to the joint layout setting, while using the output of the spectral approach as initialization. Further, we discuss a user interface for exploring a collection of graphs. Finally, we show multiple example visualizations of graphs stemming from collections of segmented meshes and we present qualitative and quantitative comparisons with previous work.

In this paper we present a novel GPU-based data structure for spatial indexing. Based on Fenwick trees-a special type of binary indexed trees-our data structure allows construction in linear time. Updates and prefixes can be computed in logarithmic time, whereas point queries require only constant time on average. Unlike competing data structures such as summed-area tables and spatial hashing, our data structure requires a constant amount of bits for each data element, and it offers unconstrained point queries. This property makes our data structure ideally suited for applications requiring unconstrained indexing of large data, such as block-storage of large and block-sparse volumes. Finally, we provide asymptotic bounds on both run-time and memory requirements, and we show applications for which our new data structure is useful.

Abstract Collaborative slide image viewing systems are becoming increasingly important in pathology applications such as telepathology and E‐learning. Despite rapid advances in computing and imaging technology, current digital pathology systems have limited performance with respect to remote viewing of whole slide images on desktop or mobile computing devices. In this paper we present a novel digital pathology client–server system that supports collaborative viewing of multi‐plane whole slide images over standard networks using multi‐touch‐enabled clients. Our system is built upon a standard HTTP web server and a MySQL database to allow multiple clients to exchange image and metadata concurrently. We introduce a domain‐specific image‐stack compression method that leverages real‐time hardware decoding on mobile devices. It adaptively encodes image stacks in a decorrelated colour space to achieve extremely low bitrates (0.8 bpp) with very low loss of image quality. We evaluate the image quality of our compression method and the performance of our system for diagnosis with an in‐depth user study.

Abstract The Closest Point Method (CPM) is a method for numerically solving partial differential equations (PDEs) on arbitrary surfaces, independent of the existence of a surface parametrization. The CPM uses a closest point representation of the surface, to solve the unmodified Cartesian version of a surface PDE in a 3D volume embedding, using simple and well‐understood techniques. In this paper, we present the numerical solution of the wave equation and the incompressible Navier‐Stokes equations on surfaces via the CPM, and we demonstrate surface appearance and shape variations in real‐time using this method. To fully exploit the potential of the CPM, we present a novel GPU realization of the entire CPM pipeline. We propose a surface‐embedding adaptive 3D spatial grid for efficient representation of the surface, and present a high‐performance approach using CUDA for converting surfaces given by triangulations into this representation. For real‐time performance, CUDA is also used for the numerical procedures of the CPM. For rendering the surface (and the PDE solution) directly from the closest point representation without the need to reconstruct a triangulated surface, we present a GPU ray‐casting method that works on the adaptive 3D grid.

We present a novel method to recycle bits of paletted data sets. We exploit that the codebook of such data can be reordered without affecting the content. Enumerating all possible permutations of N codebook entries yields an additional O(N log2N) bits that can be used-without storage overhead-for the lossless encoding of a limited amount of tags, meta-information, or part of the actual data.

Histology is the study of the structure of biological tissue using microscopy techniques. As digital imaging technology advances, high resolution microscopy of large tissue volumes is becoming feasible; however, new interactive tools are needed to explore and analyze the enormous datasets. In this paper we present a visualization framework that specifically targets interactive examination of arbitrarily large image stacks. Our framework is built upon two core techniques: display-aware processing and GPU-accelerated texture compression. With display-aware processing, only the currently visible image tiles are fetched and aligned on-the-fly, reducing memory bandwidth and minimizing the need for time-consuming global pre-processing. Our novel texture compression scheme for GPUs is tailored for quick browsing of image stacks. We evaluate the usability of our viewer for two histology applications: digital pathology and visualization of neural structure at nanoscale-resolution in serial electron micrographs.

Abstract We present a geometry compression scheme for restricted quadtree meshes and use this scheme for the compression of adaptively triangulated digital elevation models (DEMs). A compression factor of 8–9 is achieved by employing a generalized strip representation of quadtree meshes to incrementally encode vertex positions. In combination with adaptive error‐controlled triangulation, this allows us to significantly reduce bandwidth requirements in the rendering of large DEMs that have to be paged from disk. The compression scheme is specifically tailored for GPU‐based decoding, since it minimizes dependent memory access operations. We can thus trade CPU operations and CPU–GPU data transfer for GPU processing, resulting in twice faster streaming of DEMs from main memory into GPU memory. A novel storage format for decoded DEMs on the GPU facilitates a sustained rendering throughput of about 300 million triangles per second. Due to these properties, the proposed scheme enables scalable rendering with respect to the display resolution independent of the data size. For a maximum screen‐space error below 1 pixel it achieves frame rates of over 100 fps, even on high‐resolution displays. We validate the efficiency of the proposed method by presenting experimental results on scanned elevation models of several hundred gigabytes.

In this paper we investigate scalability limitations in the visualization of large-scale particle-based cosmological simulations, and we present methods to reduce these limitations on current PC architectures. To minimize the amount of data to be streamed from disk to the graphics subsystem, we propose a visually continuous level-of-detail (LOD) particle representation based on a hierarchical quantization scheme for particle coordinates and rules for generating coarse particle distributions. Given the maximal world space error per level, our LOD selection technique guarantees a sub-pixel screen space error during rendering. A brick-based page-tree allows to further reduce the number of disk seek operations to be performed. Additional particle quantities like density, velocity dispersion, and radius are compressed at no visible loss using vector quantization of logarithmically encoded floating point values. By fine-grain view-frustum culling and presence acceleration in a geometry shader the required geometry throughput on the GPU can be significantly reduced. We validate the quality and scalability of our method by presenting visualizations of a particle-based cosmological dark-matter simulation exceeding 10 billion elements.

In this paper we present GPU-based techniques for the interactive visualization of large unsteady 3D flow fields on uniform grids. We propose a novel dual-core approach to asynchronously stream such fields from the CPU, thus enabling the efficient exploration of large time-resolved sequences. This approach decouples visualization from data handling, resulting in interactive frame rates. Built upon a previously published GPU particle engine for flow visualization we have developed new strategies to compute and to visualize path lines and streak lines on the GPU. To provide additional visual cues, focus+context techniques for polygonal meshes have been integrated. The proposed techniques are used in the visual analysis of the Terashake 2.1 earthquake simulation data, and they have been shown to be very effective in revealing the relevant information in this data.

Volume rendered imagery often includes a barrage of 3D information like shape, appearance and topology of complex structures, and it thus quickly overwhelms the user. In particular, when focusing on a specific region a user cannot observe the relationship between various structures unless he has a mental picture of the entire data. In this paper we present ClearView, a GPU-based, interactive framework for texture-based volume ray-casting that allows users which do not have the visualization skills for this mental exercise to quickly obtain a picture of the data in a very intuitive and user-friendly way. ClearView is designed to enable the user to focus on particular areas in the data while preserving context information without visual clutter. ClearView does not require additional feature volumes as it derives any features in the data from image information only. A simple point-and-click interface enables the user to interactively highlight structures in the data. ClearView provides an easy to use interface to complex volumetric data as it only uses transparency in combination with a few specific shaders to convey focus and context information.

A survey of graphics developers on the issue of texture mapping hardware for volume rendering would most likely find that the vast majority of them view limited texture memory as one of the most serious drawbacks of an otherwise fine technology. In this paper, we propose a compression scheme for static and time-varying volumetric data sets based on vector quantization that allows us to circumvent this limitation. We describe a hierarchical quantization scheme that is based on a multiresolution covariance analysis of the original field. This allows for the efficient encoding of large-scale data sets, yet providing a mechanism to exploit temporal coherence in non-stationary fields. We show, that decoding and rendering the compressed data stream can be done on the graphics chip using programmable hardware. In this way, data transfer between the CPU and the graphics processing unit (GPU) can be minimized thus enabling flexible and memory efficient real-time rendering options. We demonstrate the effectiveness of our approach by demonstrating interactive renditions of Gigabyte data sets at reasonable fidelity on commodity graphics hardware.