Accurate segmentation of thoracoabdominal anatomical structures in three-dimensional medical imaging modalities is fundamental for informed clinical decision-making across a wide array of medical disciplines. Current approaches often struggle to efficiently and comprehensively process this region's intricate and heterogeneous anatomical information, leading to suboptimal outcomes in diagnosis, treatment planning, and disease management. To address this challenge, we introduce SegTom, a novel volumetric segmentation framework equipped with a cutting-edge SegTom Block specifically engineered to effectively capture the complex anatomical representations inherent to the thoracoabdominal region. This SegTom Block incorporates a hierarchical anatomical-representation decomposition to facilitate efficient information exchange by decomposing the computationally intensive self-attention mechanism and cost-effectively aggregating the extracted representations. Rigorous validation of SegTom across nine diverse datasets, encompassing both computed tomography (CT) and magnetic resonance imaging (MRI) modalities, consistently demonstrates high performance across a broad spectrum of anatomical structures. Specifically, SegTom achieves a mean Dice similarity coefficient (DSC) of 87.29% for cardiac segmentation on the MM-WHS MRI dataset, 83.48% for multi-organ segmentation on the BTCV abdominal CT dataset, and 92.01% for airway segmentation on a dedicated CT dataset.

Multimodal ultrasound imaging, combining B-mode ultrasound, shear wave velocity, and shear wave time, is crucial for diagnosing and treating breast lesions, providing insights into lesion characteristics and tissue properties. However, challenges arise from inter-modal feature misalignment and attention shifts due to varied capture methods and an overemphasis on vibrant color data. To tackle these issues, we introduce two innovations: a novel segmentation framework and a comprehensive dataset. The UltraMamba framework utilizes bidirectional alignment between modalities and enhances region-specific information to improve breast lesion segmentation accuracy. Key components include the Cross-Modal Knowledge Interaction module for robust information exchange and the Region-Aware Feature Excitation module to focus on relevant features. We also present the BreLS dataset, the first two-dimensional multimodal ultrasound breast lesion dataset, with paired images from 506 cases, serving as a valuable resource for analysis. UltraMamba shows strong performance on the BreLS dataset, achieving a Dice Similarity Coefficient of 72.16% and an HD95 of 42.02 mm, reflecting improvements of 2.59% in DSC and a 6.78 mm reduction in HD95 compared to the second-best framework, MMCA-NET. These results highlight UltraMamba's potential to enhance segmentation accuracy in clinical settings, facilitating precise treatment planning and, ultimately, leading to improved outcomes. Code: https://github.com/deepang-ai/UltraMamba.

Polyp segmentation in endoscopic imaging is essential for the early detection of colorectal cancer, as polyps are precursor lesions in the colon and rectum, yet the task is complicated by the morphological variability and indistinct boundaries of polyps, which often blend into surrounding tissues. Conventional approaches struggle with these complexities, as fixed scale and window sizes are unable to adapt to the diverse and irregular structures of polyps. To address this challenge, we introduce the Endoscopic Adaptive Transformer, EAT, a novel framework specifically engineered for polyp segmentation. EAT incorporates an adaptive perception module, APM, that employs an adaptive perceptive-field mechanism to dynamically capture both fine-grained local details and broad contextual information, enhancing segmentation accuracy across diverse polyp morphologies. EAT demonstrates comprehensive performance by achieving a Dice coefficient of 97.77% and an HD95 of 4.50mm in single-target segmentation, while also excelling in multi-target scenarios with a Dice coefficient of 88.02% and an HD95 of 53.75mm, significantly outperforming state-of-the-art methods across both single- and multi-target segmentation scenarios. This performance underscores EAT's critical role in improving the accuracy of polyp segmentation, highlighting its potential to advance diagnostic precision and treatment planning in clinical endoscopy applications. Code: https://github.com/deepang-ai/EAT.

Medical portable devices are increasingly requiring high accuracy, speed, and low inference jitter to meet the urgent demands of healthcare. Modern hybrid attention-based segmentation frameworks enhance segmentation accuracy but add complexity that can slow operational speed, complicating practical deployment in resource-limited settings. We propose Slim UNETRV2, a simplified framework that utilizes only basic convolutional operations in both the encoder and decoder, thereby reducing execution time and inference jitter. The Slim UNETRV2 block, placed in skip connections at each hierarchical stage, aggregates extracted representations and improves global processing. Experiments demonstrate that Slim UNETRV2 outperforms state-of-the-art models in terms of accuracy, speed, and inference jitter for resource-constrained medical devices. Notably, Slim UNETRV2 achieves 93.89% dice accuracy and 2.90 mm HD95 on BraTS 2021, being 16.7 times faster with only 0.225 ms of inference jitter compared to SegMamba. Code: https://github.com/deepang-ai/Slim-UNETRV2https://github.com/deepang-ai/Slim-UNETRV2.

Multimodal medical image registration critically relies on anatomical guidance for accurate spatial alignment. However, existing registration approaches fail to fully exploit the potential of segmentation labels, while joint registrationsegmentation (JRS) frameworks lack effective mechanisms for bidirectional task co-enhancement. To address these challenges, we propose CROSS-JRS, a novel framework that achieves mutual reinforcement between both tasks through synergistic innovations. Specifically, CROSS-JRS comprises three synergistic modules, each targeting a key aspect of the bidirectional interaction: The Segmentation-Guided Local-Global (SGLG) module hierarchically estimates deformations by progressively integrating multi-scale segmentation features with registration cues, while the Registration-Assisted Cross-Enhancement (RACE) module reciprocally improves segmentation accuracy through registrationderived spatial correspondences. To bridge features from complementary tasks, the Dual-Attention Domain Fusion (DADF) module aligns and fuses cross-task features via dynamic attention gating, enabling feature-level co-adaptation. Extensive experiments on two public 3D prostate MRI-US datasets demonstrate that the proposed CROSS-JRS outperforms the state-of-the-art JRS methods for both registration and segmentation tasks. The code is available at https://github.com/scuterGuoyulong/CROSS-JRS.

The self-attention mechanism of Transformer has gained considerable attention for its potential in modeling long-term temporal dependencies in electroencephalogram (EEG) signals. Despite recent advancements, Transformer-based decoding methods often neglect the explicit temporal priors inherent in EEG signals, i.e., the dependency between tokens tends to diminish as their relative temporal distance increases, which limits the efficacy of these decoding approaches. Inspired by the recent Retentive Network (RetNet), we develop a novel convolutional retentive network for EEG decoding (RetEEG), which integrates temporal priors into the self-attention mechanism to tackle the above challenge. Specifically, RetEEG incorporates a convolution module to capture local EEG features and cascades multiple bidirectional retention modules to learn global correlations within these local features. The bidirectional retention module introduces a temporal decay matrix, which imparts prior knowledge to the self-attention mechanism by accounting for variations in relative temporal distances. To validate the superiority of RetEEG, we conducted experiments on two publicly available EEG datasets. The experimental results empirically demonstrate that the proposed RetEEG achieves the state-of-the-art EEG decoding performance. Code is available at: https://github.com/kfhss/RetEEG.

Vulnerability attacks targeting smart contracts have caused significant losses of digital assets. Many approaches based on static analysis, fuzzing, and deep learning have been proposed for detecting contract vulnerabilities. However, most existing methods only support vulnerability detection within individual contracts. When contracts interact with each other through external calls, these methods fail to perform effective cross-contract security analysis, leading to false negatives and false positives. To address these limitations, we propose CrossMiner, a deep learning-based approach for vulnerability detection in contract interaction scenarios. CrossMiner enables comprehensive risk assessment for cross-contract security through trace analysis of function call chains. Specifically, CrossMiner first constructs a cross-contract dependency graph based on function call chains to effectively model inter-contract dependencies and network dynamics, and collect semantic information about contract interactions. Then, it employs a heterogeneous graph neural network with a two-level attention mechanism to finely extract and integrate complex features from the dependency graph, ultimately achieving precise risk assessment and vulnerability detection. We evaluate the effectiveness of CrossMiner on three types of smart contract vulnerabilities: reentrancy, timestamp dependency, and transaction state dependency. Experimental results demonstrate that CrossMiner achieves the best performance among all baseline methods, improving detection accuracy by 5.52%, 4.94%, and 5.60% for these vulnerabilities, and the F1 scores are improved by 5.44%, 5.02%, and 5.40%, respectively.

Computer-aided ultrasound (US) imaging is an important prerequisite for early clinical diagnosis and treatment. Due to the harsh ultrasound (US) image quality and the blurry tumor area, recent memory-based video object segmentation models (VOS) achieve frame-level segmentation by performing intensive similarity matching among the past frames which could inevitably result in computational redundancy. In this paper, we first build a larger annotated benchmark dataset for breast lesion segmentation in ultrasound videos, then we propose a lightweight clip-level VOS framework for achieving higher segmentation accuracy while maintaining the speed. Then an Inner-Outer Clip Retformer is proposed to extract spatial-temporal tumor features in parallel. Specifically, the proposed Outer Clip Retformer extracts the tumor movement feature from past video clips to locate the current clip tumor position, while the Inner Clip Retformer detailedly extracts current tumor features that can produce more accurate segmentation results. Then a Clip Contrastive loss function is further proposed to align the extracted tumor features along both the spatial-temporal dimensions to improve the segmentation accuracy. In addition, the Global Retentive Memory is proposed to maintain the complementary tumor features with lower computing resources which can generate coherent temporal movement features. In this way, our model can significantly improve the spatial-temporal perception ability without increasing a large number of parameters, achieving more accurate segmentation results while maintaining a faster segmentation speed. Finally, we conduct extensive experiments to evaluate our proposed model on several video object segmentation datasets, the results show that our framework outperforms state-of-the-art segmentation methods.

Medical video segmentation is fundamentally important in clinical diagnosis and treatment procedures, offering dynamic tracking of breast lesions across frames in ultrasound videos for improved segmentation performance. However, existing approaches face challenges in striking a balance between segmentation performance and inference speed, hindering real-time application in resource-constrained medical environments. In order to address these limitations, we present BaS, a blazing-fast on-device breast lesion segmentation model. BaS integrates the Stem module and BaSBlock to refine representations through inter- and intra-frame analysis on ultrasound videos. In addition, we release two versions of BaS: the BaS-S for superior segmentation performance and the BaS-L for accelerated inference times. Experimental Results indicate that BaS surpasses the top-performing models in terms of segmenting efficiency and accuracy of predictions on devices with limited resources. This work advances the development of efficient medical video segmentation frameworks applicable to multiple medical platforms.

In the specialized domain of brain tumor segmentation, supervised segmentation approaches are hindered by the limited availability of high-quality labeled data, a condition arising from data privacy concerns, significant costs, and ethical issues. In response to this challenge, this paper presents a training framework that adeptly integrates a plug-and-play component, MOD, into current supervised learning models, boosting their efficacy in scenarios with limited data. The MOD consists of an Online Tokenizer and a Dense Predictor, which employs self-distillation and self-modeling on masked patches, promoting swift convergence and efficient representation learning. During the inference phase, the plug-and-play MOD component is excluded, preserving the computational efficiency of the original model without incurring extra processing costs. We substantiated the value of our approach through experiments on leading 3D brain tumor segmentation baselines. Remarkably, models augmented with the MOD consistently showcased superior results, achieving elevated Dice coefficients and HD95 scores on two datasets: BraTS 2021 and MSD 2019 Task-01 Brain Tumor.

Automatic report generation has arisen as a significant research area in computer-aided diagnosis, aiming to alleviate the burden on clinicians by generating reports automatically based on medical images. In this work, we propose a novel framework for automatic ultrasound report generation, leveraging a combination of unsupervised and supervised learning methods to aid the report generation process. Our framework incorporates unsupervised learning methods to extract potential knowledge from ultrasound text reports, serving as the prior information to guide the model in aligning visual and textual features, thereby addressing the challenge of feature discrepancy. Additionally, we design a global semantic comparison mechanism to enhance the performance of generating more comprehensive and accurate medical reports. To enable the implementation of ultrasound report generation, we constructed three large-scale ultrasound image-text datasets from different organs for training and validation purposes. Extensive evaluations with other state-of-the-art approaches exhibit its superior performance across all three datasets. Code and dataset are valuable at this link.

Masked Autoencoder (MAE) is a self-supervised pre-training technique that holds promise in improving the representation learning of neural networks. However, the current application of MAE directly to volumetric medical images poses two challenges: (i) insufficient global information for clinical context understanding of the holistic data, and (ii) the absence of any assurance of stabilizing the representations learned from randomly masked inputs. To conquer these limitations, we propose the Global-Local Masked AutoEncoders (GL-MAE), a simple yet effective self-supervised pre-training strategy. GL-MAE acquires robust anatomical structure features by incorporating multi-level reconstruction from fine-grained local details to high-level global semantics. Furthermore, a complete global view serves as an anchor to direct anatomical semantic alignment and stabilize the learning process through global-to-global consistency learning and global-to-local consistency learning. Our fine-tuning results on eight mainstream public datasets demonstrate the superiority of our method over other state-of-the-art self-supervised algorithms, highlighting its effectiveness on versatile volumetric medical image segmentation and classification tasks. We will release codes upon acceptance at https://github.com/JiaxinZhuang/GL-MAE.

The Vision Transformer (ViT) has demonstrated remarkable performance in Self-Supervised Learning (SSL) for 3D medical image analysis. Masked AutoEncoder (MAE) for feature pre-training can further unleash the potential of ViT on various medical vision tasks. However, due to large spatial sizes with much higher dimensions of 3D medical images, the lack of hierarchical design for MAE may hinder the performance of downstream tasks. In this paper, we propose a novel Mask in Mask (MiM) pre-training framework for 3D medical images, which aims to advance MAE by learning discriminative representation from hierarchical visual tokens across varying scales. We introduce multiple levels of granularity for masked inputs from the volume, which are then reconstructed simultaneously ranging at both fine and coarse levels. Additionally, a cross-level alignment mechanism is applied to adjacent level volumes to enforce anatomical similarity hierarchically. Furthermore, we adopt a hybrid backbone to enhance the hierarchical representation learning efficiently during the pre-training. MiM was pre-trained on a large scale of available 3D volumetric images, i.e., Computed Tomography (CT) images containing various body parts. Extensive experiments on twelve public datasets demonstrate the superiority of MiM over other SSL methods in organ/tumor segmentation and disease classification. We further scale up the MiM to large pre-training datasets with more than 10k volumes, showing that large-scale pre-training can further enhance the performance of downstream tasks. Code is available at https://github.com/JiaxinZhuang/MiM.

This paper investigates an intriguing yet unsolved problem of cross-scene background subtraction for training only one deep model to process large-scale video streaming. We propose an end-to-end cross-scene background subtraction network via 3D optical flow, dubbed CrossNet. First, we design a new motion descriptor, hierarchical 3D optical flows (3D-HOP), to observe fine-grained motion. Then, we build a cross-modal dynamic feature filter (CmDFF) to enable the motion and appearance feature interaction. CrossNet exhibits better generalization since the proposed modules are encouraged to learn more discriminative semantic information between the foreground and the background. Furthermore, we design a loss function to balance the size diversity of foreground instances since small objects are usually missed due to training bias. Our whole background subtraction model is called Hierarchical Optical Flow Attention Model (HOFAM). Unlike most of the existing stochastic-process-based and CNN-based background subtraction models, HOFAM will avoid inaccurate online model updating, not heavily rely on scene-specific information, and well represent ambient motion in the open world. Experimental results on several well-known benchmarks demonstrate that it outperforms state-of-the-art by a large margin. The proposed framework can be flexibly integrated into arbitrary streaming media systems in a plug-and-play form. Codes are available athttps://github.com/dongzhang89/HOFAM.

The widely deployed ways to capture a set of unorganized points, e.g., merged laser scans, fusion of depth images, and structure-from- , usually yield a 3-D noisy point cloud. Accurate normal estimation for the noisy point cloud makes a crucial contribution to the success of various applications. However, the existing normal estimation wisdoms strive to meet a conflicting goal of simultaneously performing normal filtering and preserving surface features, which inevitably leads to inaccurate estimation results. We propose a normal estimation neural network (Norest-Net), which regards normal filtering and feature preservation as two separate tasks, so that each one is specialized rather than traded off. For full noise removal, we present a normal filtering network (NF-Net) branch by learning from the noisy height map descriptor (HMD) of each point to the ground-truth (GT) point normal; for surface feature recovery, we construct a normal refinement network (NR-Net) branch by learning from the bilaterally defiltered point normal descriptor (B-DPND) to the GT point normal. Moreover, NR-Net is detachable to be incorporated into the existing normal estimation methods to boost their performances. Norest-Net shows clear improvements over the state of the arts in both feature preservation and noise robustness on synthetic and real-world captured point clouds.

Emotion classification based on electroencephalogram (EEG) signals has drawn huge attention in affective brain computer interface (BCI). Recently, plenty of deep learning approaches have been proposed to improve the performance of EEG emotion recognition, especially the application of domain adaptation methods to tackle the challenge of large individual differences of EEG signals from subject to subject. However, these conventional transfer learning methods would result in information leakage during the sharing of domain data to enhance the accuracy of the target tasks. Therefore, in this paper, we proposed a distributed deep learning method, named adaptive federated learning (AdaFL) for EEG emotion recognition. In AdaFL, a server collaboratively learns a global model by adaptively aggregating the local models according to their importance in several communication rounds. In particular, an importance function is developed to evaluate each client, which would determine to select a subset of optimal clients for subsequent global model aggregation. The function is a simple transformation of the training loss and the sample size of the local models. Then, the resulting importance scores of selected clients are further converted into aggregation coefficients to measure the weights of the local models for global model aggregation. The distinct advantage of AdaFL is that the cross-subject information could be well utilized and the information leakage risk could be significantly reduced. To validate the efficacy of the proposed AdaFL, we conduct extensive experiments on two real EEG emotion datasets, i.e., SEED and DEAP. The experimental results show that the proposed AdaFL has achieved 94.95 ± 0.40% and 91.06 ± 0.29% average classification accuracy on the SEED and DEAP datasets, respectively, which reflects the superiority of our method over the state-of-the-art approaches.

Video Shadow Detection (VSD) aims to detect the shadow masks with frame sequence. Existing works suffer from inefficient temporal learning. Moreover, few works address the VSD problem by considering the characteristic (i.e., boundary) of shadow. Motivated by this, we propose a Timeline and Boundary Guided Diffusion (TBGDiff) network for VSD where we take account of the past-future temporal guidance and boundary information jointly. In detail, we design a Dual Scale Aggregation (DSA) module for better temporal understanding by rethinking the affinity of the long-term and short-term frames for the clipped video. Next, we introduce Shadow Boundary Aware Attention (SBAA) to utilize the edge contexts for capturing the characteristics of shadows. Moreover, we are the first to introduce the Diffusion model for VSD in which we explore a Space-Time Encoded Embedding (STEE) to inject the temporal guidance for Diffusion to conduct shadow detection. Benefiting from these designs, our model can not only capture the temporal information but also the shadow property. Extensive experiments show that the performance of our approach overtakes the state-of-the-art methods, verifying the effectiveness of our components. We release the codes, weights, and results at \url{https://github.com/haipengzhou856/TBGDiff}.

Masked Image Modeling (MIM) has achieved impressive representative performance with the aim of reconstructing randomly masked images. Despite the empirical success, most previous works have neglected the important fact that it is unreasonable to force the model to reconstruct something beyond recovery, such as those masked objects. In this work, we show that uniformly random masking widely used in previous works unavoidably loses some key objects and changes original semantic information, resulting in a misalignment problem and hurting the representative learning eventually. To address this issue, we augment MIM with a new masking strategy namely the DPPMask by substituting the random process with Determinantal Point Process (DPPs) to reduce the semantic change of the image after masking. Our method is simple yet effective and requires no extra learnable parameters when implemented within various frameworks. In particular, we evaluate our method on two representative MIM frameworks, MAE and iBOT. We show that DPPMask surpassed random sampling under both lower and higher masking ratios, indicating that DPP-Mask makes the reconstruction task more reasonable. We further test our method on the background challenge and multi-class classification tasks, showing that our method is more robust at various tasks.

Puncture robots pave a new way for stable, accurate and safe percutaneous liver tumor puncture operation. However, affected by respiratory motion, intraoperative accurate location of the tumor and its surrounding anatomical structures remains a difficult problem in existing robot-assisted puncture operations. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D ultrasound (US) and preoperative 3D computed tomography (CT) fusion is proposed, addressing the shortcomings of existing puncture robots. To deal with the challenge of cross-modal and cross-dimensional registration between 2D US and 3D CT, a decoupled two-stage registration approach combining initial vessel structure-based 3D US – 3D CT registration with intraoperative intensity-based 2D US -3D US registration is proposed. To achieve fast and robust ultrasound probe calibration, a method based on an improved N-wire phantom is proposed. Twenty puncture experiments are performed in different breath-holding positions on a respiratory motion simulation platform, and experimental results show that the mean puncture error is 2.48 mm, which can meet the requirements in a wide of clinical scenariosNote to Practitioners—In clinical percutaneous liver tumor puncture operation, due to the lack of real-time and clear image guidance, it is difficult to locate the tumor and its surrounding vital anatomical structures. In addition, the stability and accuracy of manual operation are poor. The development of a puncture robot is an effective solution for these problems. However, existing CT and magnetic resonance imaging (MRI) guided robots do not consider the tumor localization errors caused by inconsistent breath-holding positions between preoperative scan period and intraoperative puncture period, and US guided robots are limited by the poor image quality and the narrow field of vision. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D US and preoperative 3D CT fusion is proposed. This system can take advantage of the real-time ultrasound and clear CT images at the same time, and can provide real-time, clear and all-round guidance for percutaneous liver tumor puncture operation, which has obvious advantages over the existing puncture robots. Phantom experiments have been completed and animal experiments will be carried out in the future.

Hybrid transformer-based segmentation approaches have shown great promise in medical image analysis. However, they typically require considerable computational power and resources during both training and inference stages, posing a challenge for resource-limited medical applications common in the field. To address this issue, we present an innovative framework called Slim UNETR, designed to achieve a balance between accuracy and efficiency by leveraging the advantages of both convolutional neural networks and transformers. Our method features the Slim UNETR Block as a core component, which effectively enables information exchange through self-attention mechanism decomposition and cost-effective representation aggregation. Additionally, we utilize the throughput metric as an efficiency indicator to provide feedback on model resource consumption. Our experiments demonstrate that Slim UNETR outperforms state-of-the-art models in terms of accuracy, model size, and efficiency when deployed on resource-constrained devices. Remarkably, Slim UNETR achieves 92.44% dice accuracy on BraTS2021 while being 34.6x smaller and 13.4x faster during inference compared to Swin UNETR. Code: https://github.com/aigzhusmart/Slim-UNETR.

Kinesthetic feedback, the feeling of restriction or resistance when hands contact objects, is essential for natural freehand interaction in VR. However, inducing kinesthetic feedback using mechanical hardware can be cumbersome and hard to control in commodity VR systems. We propose the kine-appendage concept to compensate for the loss of kinesthetic feedback in virtual environments, i.e., a virtual appendage is added to the user's avatar hand; when the appendage contacts a virtual object, it exhibits transformations (rotation and deformation); when it disengages from the contact, it recovers its original appearance. A proof-of-concept kine-appendage technique, BrittleStylus, was designed to enhance isomorphic typing. Our empirical evaluations demonstrated that (i) BrittleStylus significantly reduced the uncorrected error rate of naive isomorphic typing from 6.53% to 1.92% without compromising the typing speed; (ii) BrittleStylus could induce the sense of kinesthetic feedback, the degree of which was parity with that induced by pseudo-haptic (+ visual cue) methods; and (iii) participants preferred BrittleStylus over pseudo-haptic (+ visual cue) methods because of not only good performance but also fluent hand movements.

When capturing and storing images, devices inevitably introduce noise. Reducing this noise is a critical task called image denoising. Deep learning has become the de facto method for image denoising, especially with the emergence of Transformer-based models that have achieved notable state-of-the-art results on various image tasks. However, deep learning-based methods often suffer from a lack of generalization ability. For example, deep models trained on Gaussian noise may perform poorly when tested on other noise distributions. To address this issue, we present a novel approach to enhance the generalization performance of denoising networks, known as masked training. Our method involves masking random pixels of the input image and reconstructing the missing information during training. We also mask out the features in the self-attention layers to avoid the impact of training-testing inconsistency. Our approach exhibits better generalization ability than other deep learning models and is directly applicable to real-world scenarios. Additionally, our interpretability analysis demonstrates the superiority of our method.

In biological research, fluorescence staining is a key technique to reveal the locations and morphology of subcellular structures. However, it is slow, expensive, and harmful to cells. In this paper, we model it as a deep learning task termed subcellular structure prediction (SSP), aiming to predict the 3D fluorescent images of multiple subcellular structures from a 3D transmitted-light image. Unfortunately, due to the limitations of current biotechnology, each image is partially labeled in SSP. Besides, naturally, subcellular structures vary considerably in size, which causes the multi-scale issue of SSP. To overcome these challenges, we propose Re-parameterizing Mixture-of-Diverse-Experts (RepMode), a network that dynamically organizes its parameters with task-aware priors to handle specified single-label prediction tasks. In RepMode, the Mixture-of-Diverse-Experts (MoDE) block is designed to learn the generalized parameters for all tasks, and gating re-parameterization (GatRep) is performed to generate the specialized parameters for each task, by which RepMode can maintain a compact practical topology exactly like a plain network, and meanwhile achieves a powerful theoretical topology. Comprehensive experiments show that RepMode can achieve state-of-the-art overall performance in SSP.

Approximate computing has been widely used in many fault-tolerant applications. Multiplication as a key kernel in such applications, it is significant to improve the efficiency of approximate multiplier to achieve high computational performance. This paper proposes a novel approximate multiplier design based on using different compressors for different regions of partial products. We designed two Preprocessing Units (PUs) to explore the best efficiency via increasing the number of sparse partial products. Multiple 8-bit multipliers are designed using Verilog and synthesized under the 45-nm CMOS technology. Compared with the conventional Wallace Tree multiplier, experimental results indicate that one of our proposed multipliers reduce Power-Delay Product (PDP) by 58.5% at most with 0.42% normalized mean error distance. Moreover, a case study of image processing applications is also investigated. Our proposed multipliers can achieve a high peak signal-to-noise ratio of 51.87dB. Compared to the state-of-the-art, the proposed multiplier has a better comprehensive performance in accuracy, area and power consumption.

Monocular Depth Estimation (MDE) is a fundamental task in computer vision and multimedia. With the wide applications of deep Convolutional Neural Networks (CNNs), learning-based methods have achieved superior performance on MDE tasks in recent years. Because loss functions are important to train an accurate CNN with good generalization performance, nearly all previous efforts contribute to proposing powerful loss functions with careful hand-crafted regularizers(e.g., gradient loss and normal loss) added to the basic depth L1-Loss. However, the hand-crafted regularizers require rich domain knowledge, while their performance can still not be guaranteed. In this paper, we learn a new regularizer, approximated by a tiny CNN Regularizrer-Net(RN), and train it in an adversarial way. As demonstrated experimentally, our learned regularizer can notably outperform the current state-of-the-art methods by both quantitative evaluation and qualitative visualization on the benchmark NYU-Depth-v2 dataset, and well generalize to the new ScanNet dataset without any further training. Our code will be released soon.

The backpropagation networks are notably susceptible to catastrophic forgetting, where networks tend to forget previously learned skills upon learning new ones. To address such the 'sensitivity-stability' dilemma, most previous efforts have been contributed to minimizing the empirical risk with different parameter regularization terms and episodic memory, but rarely exploring the usages of the weight loss landscape. In this paper, we investigate the relationship between the weight loss landscape and sensitivity-stability in the continual learning scenario, based on which, we propose a novel method, Flattening Sharpness for Dynamic Gradient Projection Memory (FS-DGPM). In particular, we introduce a soft weight to represent the importance of each basis representing past tasks in GPM, which can be adaptively learned during the learning process, so that less important bases can be dynamically released to improve the sensitivity of new skill learning. We further introduce Flattening Sharpness (FS) to reduce the generalization gap by explicitly regulating the flatness of the weight loss landscape of all seen tasks. As demonstrated empirically, our proposed method consistently outperforms baselines with the superior ability to learn new skills while alleviating forgetting effectively.

Shadow detection in general photos is a nontrivial problem, due to the complexity of the real world. Though recent shadow detectors have already achieved remarkable performance on various benchmark data, their performance is still limited for general real-world situations. In this work, we collected shadow images for multiple scenarios and compiled a new dataset of 10,500 shadow images, each with labeled ground-truth mask, for supporting shadow detection in the complex world. Our dataset covers a rich variety of scene categories, with diverse shadow sizes, locations, contrasts, and types. Further, we comprehensively analyze the complexity of the dataset, present a fast shadow detection network with a detail enhancement module to harvest shadow details, and demonstrate the effectiveness of our method to detect shadows in general situations.

Instance shadow detection is a brand new problem, aiming to find shadow instances paired with object instances. To approach it, we first prepare a new dataset called SOBA, named after Shadow-OBject Association, with 3,623 pairs of shadow and object instances in 1,000 photos, each with individual labeled masks. Second, we design LISA, named after Light-guided Instance Shadow-object Association, an end-to-end framework to automatically predict the shadow and object instances, together with the shadow-object associations and light direction. Then, we pair up the predicted shadow and object instances, and match them with the predicted shadow-object associations to generate the final results. In our evaluations, we formulate a new metric named the shadow-object average precision to measure the performance of our results. Further, we conducted various experiments and demonstrate our method's applicability on light direction estimation and photo editing.

In this paper, we propose a normal estimation method for unstructured point cloud. We observe that geometric estimators commonly focus more on feature preservation but are hard to tune parameters and sensitive to noise, while learning-based approaches pursue an overall normal estimation accuracy but cannot well handle challenging regions such as surface edges. This paper presents a novel normal estimation method, under the co-support of geometric estimator and deep learning. To lowering the learning difficulty, we first propose to compute a suboptimal initial normal at each point by searching for a best fitting patch. Based on the computed normal field, we design a normal-based height map network (NH-Net) to fine-tune the suboptimal normals. Qualitative and quantitative evaluations demonstrate the clear improvements of our results over both traditional methods and learning-based methods, in terms of estimation accuracy and feature recovery.

This paper presents a novel mixed reality based navigation system for accurate respiratory liver tumor punctures in radiofrequency ablation (RFA). Our system contains an optical see-through head-mounted display device (OST-HMD), Microsoft HoloLens for perfectly overlaying the virtual information on the patient, and a optical tracking system NDI Polaris for calibrating the surgical utilities in the surgical scene. Compared with traditional navigation method with CT, our system aligns the virtual guidance information and real patient and real-timely updates the view of virtual guidance via a position tracking system. In addition, to alleviate the difficulty during needle placement induced by respiratory motion, we reconstruct the patient-specific respiratory liver motion through statistical motion model to assist doctors precisely puncture liver tumors. The proposed system has been experimentally validated on vivo pigs with an accurate real-time registration approximately 5-mm mean FRE and TRE, which has the potential to be applied in clinical RFA guidance.

This paper presents a novel augmented reality (AR) interactive environment for neurosurgical training. Comparing with traditional virtual reality based neurosurgical simulator, our system provides a more natural and intuitive fashion for surgeons. To achieve holographic visualization of virtual brain on 3D-printed skull (workspace), the first step is to reconstruct the personalized anatomy structure from segmented MR imaging. Then, tailored to the computational power of HoloLens, we employ the mass-spring method to model the mechanical response of brain. After that, a precise registration method is employed to map the virtual-real spatial information, which can overlay the virtual operative brain on workspace. In addition, bimanual haptic interface is also integrated into our simulator, which is more similar with real neurosurgery. In experiments, we conduct accuracy validation on our registration method, as well as the validity test on the developed simulators. The results demonstrate that our simulator can provide high-accuracy augmented visualization effects and deep immersion for novice surgeons.

Realistic liquid animations with thin sheets or streams are crucial for creating fluid effects in digital media. However, it is challenging to simulate these appealing thin sheets or streams in the framework of smoothed particle hydrodynamics (SPH). The underlying reason for this challenge mainly lies in the inherent numerical instability of SPH due to inconsistent kernel interpolation, which is caused by the incomplete kernel support on the free surface and the particles' disorder dispersion within the simulation domain. To address this challenge, we propose a novel and effective approach to ensure the consistency of kernel interpolation at both internal flow and the free surface during the simulation such that these thin features can always be well maintained. First, we introduce a transport-velocity formulation to alleviate the disorder dispersion in the liquid domain. However, this formulation can only work in the internal flow, and it fails at the free surface because it cannot accurately estimate the density of particles there. To this end, we propose adaptively correcting the underestimated density caused by the incomplete kernel support of free-surface particles, which are identified by a geometry-aware anisotropic kernel, to counteract the inconsistent interpolation on the free surface. Then, we propose a novel scheme to further filter the background pressure to enhance the interactions between the internal flow and the free surface, as well as liquid and solid, such that the thin features generated from such interactions can be realistically simulated. The proposed approach can also achieve anticlumping and regularization effects in the entire simulation domain and, hence, further enhance the thin features in liquids. We evaluate our method on a variety of benchmark examples, and the results demonstrate that our method can achieve more appealing visual effects than state-of-the-art methods by realistically simulating more vivid thin features.

Turbulent vortices in smoke flows are crucial for a visually interesting appearance. Unfortunately, it is challenging to efficiently simulate these appealing effects in the framework of vortex filament methods. The vortex filaments in grids scheme allows to efficiently generate turbulent smoke with macroscopic vortical structures, but suffers from the projection-related dissipation, and thus the small-scale vortical structures under grid resolution are hard to capture. In addition, this scheme cannot be applied in wall-bounded turbulent smoke simulation, which requires efficiently handling smoke-obstacle interaction and creating vorticity at the obstacle boundary. To tackle above issues, we propose an effective filament-mesh particle-particle (FMPP) method for fast wall-bounded turbulent smoke simulation with ample details. The Filament-Mesh component approximates the smooth long-range interactions by splatting vortex filaments on grid, solving the Poisson problem with a fast solver, and then interpolating back to smoke particles. The Particle-Particle component introduces smoothed particle hydrodynamics (SPH) turbulence model for particles in the same grid, where interactions between particles cannot be properly captured under grid resolution. Then, we sample the surface of obstacles with boundary particles, allowing the interaction between smoke and obstacle being treated as pressure forces in SPH. Besides, the vortex formation region is defined at the back of obstacles, providing smoke particles flowing by the separation particles with a vorticity force to simulate the subsequent vortex shedding phenomenon. The proposed approach can synthesize the lost small-scale vortical structures and also achieve the smoke-obstacle interaction with vortex shedding at obstacle boundaries in a lightweight manner. The experimental results demonstrate that our FMPP method can achieve more appealing visual effects than vortex filaments in grids scheme by efficiently simulating more vivid thin turbulent features.

Motor imagery (MI) based on brain computer interfaces (BCIs) have been widely applied for upper limb motor rehabilitation. Due to the fact that a large number of disabled people need to restore or improve walking ability, it is also important to investigate the use of MI-based BCIs for lower limb motor rehabilitation. The brain activity of lower limb MI is more difficult to detect because of low reliability. The purpose of this study is to find a suitable paradigm of walking imagery to achieve better training effect and ensure reliable brain activity. We developed the text-based paradigm and the virtual environment (VE)-based paradigm, and evaluated their performance on identifying walking imagery from idle state.The experimental results provide evidences that the VE-based paradigm could improve the average classification accuracy. This paradigm would induce EEG patterns that make them easier for single-trial detection of walking imagery. This study has the potential to improve the reliability and robustness of walking imagery based BCIs.

Virtual surgery is a serious game which provides an opportunity to acquire cognitive and technical surgical skills via virtual surgical training and planning. However, interactively and realistically manipulating the human organ and simulating its motion under interaction is still a challenging task in this field. The underlying reason for this issue is the conflict requirements for physical constraints with high fidelity and real-time performance. To achieve realistic simulation of human organ motion with volume conservation, smooth interpolation under large deformation and precise frictional contact mechanics of global behavior in surgical scenario. This paper presents a novel and effective patch Green coordinates based interpolation for embedded deformable model to achieve the volume-preserving and smooth interpolation effects. Besides, we resolve the frictional contact mechanics for embedded deformable model, and further provide the precise boundary conditions for mechanical solver. In addition, our embedded deformable model is based on the total lagrangian explicit dynamics (TLED) finite element method (FEM) solver, which can well handle the large biological tissue deformation with both nonlinear geometric and material properties. In real compression experiments, our method can achieve liver deformation with average accuracy of 3.02 mm. Besides, the experimental results demonstrate that our method can also achieve smoother interpolation and volume-preserving effects than original embedded deformable model, and allows complex and accurate organ motion with mechanical interactions in virtual surgery.

Uterine fibroids segmentation in ultrasound images is of great importance in the definition of intra‐operative planning of ultrasound‐guided high‐intensity focused ultrasound (HIFU) therapy. However, it is challenging to obtain accurate, robust and efficient uterine fibroid segmentation due to low quality of ultrasound images. In this study, the authors propose a novel adaptive localised region and edge‐based active contour model using shape constraint and sub‐global information to accurately and efficiently segment the uterine fibroids in ultrasound images with robustness against initial contour. The authors first define adaptive local radius for the localised region‐based model and combine it with the edge‐based model to accurately and efficiently capture image's heterogeneous features and edge features. Then, they incorporate a shape constraint to reduce boundary leakage or excessive contraction to obtain more accurate segmentation. To overcome the initialisation sensitivity, they introduce the sub‐global information to prevent the curve from trapping into the local minima and obtain robust results. Furthermore, the authors optimise computation by adaptively sharing local region and employing the multi‐scale segmentation method to achieve efficient segmentation. The proposed method is validated by uterine fibroid ultrasound images in HIFU therapy and the results demonstrate that it can achieve accurate, robust and efficient segmentation.

Abstract Turbulent wake is crucial for the visually appealing effects of liquid. Unfortunately, it is challenging to realistically simulate this phenomenon with ring‐shaped vortical structures. To tackle this issue, we propose a filament‐based turbulent wake synthesis method for realistically simulating the turbulent wake with ring‐shaped vortical structures. The filaments are sampled at the separation points on the obstacle surface and emitted into the liquid flow to generate structured turbulent wake. Besides, the surface tension model is incorporated to generate natural turbulent wake diffusion visual effects in liquid by the anticurvature effects. The proposed approach can realistically and effectively synthesize the turbulent wake with ring‐shaped vortical structures and make it diffuse naturally. The experimental results demonstrate that our method outperforms than the vortex particle‐based method in synthesizing appealing turbulent wake.

In our work, a virtual reality-based surgical simulator for the mandibular angle reduction was designed and implemented on CUDA-based platform. High-fidelity visual and haptic feedbacks between the surgical instruments and the bone material are provided to enhance the perception in a realistic virtual surgical environment. Impulse-based dynamics haptic model was employed to simulate the contact forces generated on the high-speed instruments, including the reciprocating saw and the round burr. The validity of the simulated contact forces was verified by comparing against the actual force data measured through the constructed mechanical platform. An empirical study based on the patient specified data was conducted to evaluate the ability of the proposed system in training surgeons with various experiences. The results confirm the validity of our simulator.

Mandibular angle reduction is a popular and efficient procedure widely used to alter the facial contour. The primary surgical instruments, the reciprocating saw and the round burr, employed in the surgery have a common feature: operating at a high-speed. Generally, inexperienced surgeons need a long-time practice to learn how to minimize the risks caused by the uncontrolled contacts and cutting motions in manipulation of instruments with high-speed reciprocation or rotation. A virtual reality-based surgical simulator for the mandibular angle reduction was designed and implemented on a CUDA-based platform in this paper. High-fidelity visual and haptic feedbacks are provided to enhance the perception in a realistic virtual surgical environment. The impulse-based haptic models were employed to simulate the contact forces and torques on the instruments. It provides convincing haptic sensation for surgeons to control the instruments under different reciprocation or rotation velocities. The real-time methods for bone removal and reconstruction during surgical procedures have been proposed to support realistic visual feedbacks. The simulated contact forces were verified by comparing against the actual force data measured through the constructed mechanical platform. An empirical study based on the patient-specific data was conducted to evaluate the ability of the proposed system in training surgeons with various experiences. The results confirm the validity of our simulator.

Currently, electronic literature becomes increasing common for its convenient and informative features, however, the sharp increase in the number of literature has brought certain difficulties for the users to manage and access to such literatures. This demand is satisfied to some extent by the literature search engines on some existing well-known websites. But such literature search does not take into account various elements thus the access efficiency is far from the expectation. To address this issue, this paper, combining with the demands of research institutes, teams and other small groups, has developed a literature search algorithm based on AHP (Analytic Hierarchy Process). This system fully considers all the aspects of elements and improves the literature search algorithm by applying analytic hierarchy model, thus enables the ranking of literature search results to be much more valuable and beneficial to the users. Research and testing have shown that, compared with Google Scholar, the proposed algorithm in this paper is more accurate, advanced and effective in accessing and managing literatures for the researchers in academic institutes.