Semi-supervised learning (SSL) provides an effective solution to the problem of requiring time-consuming annotations in medical image segmentation. In this work, we propose a novel semi-supervised learning network(PMVA) based on multi-view annotation for medical image segmentation. Specifically, we propose the multi-view annotation by only labeling three multi-view slices in a labeled volume, which significantly relieves the burden of annotation. Then, we perform registration to obtain the initial pseudo labels for sparsely labeled volumes. Subsequently, by introducing unlabeled volumes, we propose a triplet-network paradigm that exploits dense pseudo labels in the early stage and sparse labels in the later stage and meanwhile forces the consistent output of three networks. Finally, the experimental results both on the medical data of Computer Tomography (CT) and Magnetic Resonance Imaging (MRI) in the open clinical library demonstrate that our method can generate effective pseudo labels from the labeled medical images to the unlabeled ones efficiently. Compared with existing algorithms, comprehensive quantitative and qualitative evaluations have shown excellent segmentation performance of our method.

Deep learning-based medical segmentation methods suffer from the limited amount of data due to data privacy and ethical issues, which results in generalization degradation and hinders their applications in clinical scenarios. In this paper, we propose a novel data augmentation method named Adversarial Multi-Sample Interpolation (AMSI) based on adaptive interpolations among multiple latent codes to produce new data samples. AMSI first introduces an AutoDecoder to construct the latent codes for given data points and then augment the latent space through learning-based adaptive interpolations. Specifically, we design a multi-dimension interpolation among multiple samples with learnable coefficients to adaptively generate more diverse hard examples via adversarial training. Furthermore, our method generates both synthesized images and pseudo labels simultaneously, which is different against previous style-augmented methods. Additionally, the adjacent relationship between latent points will be considered for the sake of realistic appearance and smooth pseudo labels. The effectiveness and generalization ability of AMSI are validated by extensive experiments both on two segmentation tasks employing four publicly available datasets.

Semi-supervised learning alleviates the problem of lack of labeled data, which has attracted wide attention in the field of medical image segmentation. In this paper, we propose a multi-scale prototype constrained architecture with relation aggregation for semi-supervised medical image segmentation. Concretely, we design the multi-scale prototype constraints such that the ground truth of labeled data can guide the segmentation of unlabeled data through the prototype. Furthermore, to better obtain the related prior knowledge from the annotation information, we designed the relation aggregation module, which can transmit relevant information between the labeled and unlabeled data. Comprehensive quantitative and qualitative evaluations show that our method is a general and efficient method for semi-supervised medical image segmentation.

We design a lightweight CNN, a practical and inexpensive scheme to reconstruct the geometry of the medical target from a 2D medical sequence. We make full use of natural images with rich texture features to compensate for the low-quality problem of fewer texture conditions in medical images. Besides, we design a Non-rigid Structure from Motion(NRSfM) scheme to estimate the camera structure and pose of consecutive frames with the view synthesis as the supervisory signal. Specifically, for achieving 3D reconstruction with a better time performance, we compress the CNN and medical image 3D prediction can be achieved in the lightweight CNN with the model size of 18M. The experiment results from LITFL of the open clinical library, as well as the Synapse multi-organ segmentation dataset, show that the proposed method can reconstruct the accurate geometry with better time performance.

3D reconstruction of medical images is required for many clinical scenarios as the aided diagnosis. In this study, we propose a novel medical image 3D reconstruction framework based on a transformer-based deep learning model, in which non-rigid Structure from Motion (NRSfM) is used to estimate the non-rigid deformation, and coplanar constraint is considered for the finer reconstruction result. We design a 3-stage feature extractor transformer as the backbone and take a multitask output structure to predict the photometric parameters of depth, pose, and camera structure. In addition, to obtain robust features, we pre-learn the features from the natural images with rich texture and transfer the knowledge to medical image learning. The experimental for both computed tomography (CT) and ultrasound images from the open clinical libraries show that our method can efficiently estimate the camera structure and motion, and the more precise 3D reconstruction can be achieved.

The problem of 3D reconstruction of medical images is very challenging due to the disadvantages of less texture and serious speckle noise. In this paper, we propose a medical image 3D reconstruction framework based on transfer learning and Structure-from-Motion. Unlike some existing methods, which need special hardware devices, or be effective for a particular modal of images, our unified framework does not depend on any special equipment and can be applied to diverse modalities of medical images, such as ultrasonic image, computed tomography (CT) etc. Its contribution covers three aspects: First, it uses Structure-From-Motion(SFM) to estimate the camera structure and pose of consecutive frames end-to-end, which is suitable for clinical auxiliary diagnosis. Second, strong and robust photogrammetry knowledge, including depth, the camera structure and pose, can be pre-learned from the natural images with the richer texture, and then, are transferred to the medical images learning. Besides, Med-3D is proposed, a GAN model based on self-supervised learning framework, in the generator, the spatial coplanar characteristics to constrain the cloud points reconstructed from a single captured frame or adjacent frames are considered for learning more precise features. Experimental results on the medical data of ultrasound images, and CT in the open clinical library, demonstrate that our method can reconstruct geometric structures from 2D slices and has excellent accuracy in terms of reconstruction effects.

The combination of convolution and Transformer applied to medical image segmentation has achieved great success. However, it still cannot reach extremely accurate segmentation on complex and low-contrast anatomical structures under lower calculation. To solve this problem, we propose ECT-NAS method to automatically search Efficient CNN-Transformers architecture for medical image segmentation, which featured with multi-scale search space. To better extract the global context in the search space of ECT-NAS, we carefully design a light transformer with local-global attention. Last, we proposed an efficient resource constrained search strategy that simultaneously optimizes the accuracy and cost (Params/FLOP) of the model. We evaluate ECT-NAS by conducting extensive experiments on synapse multi-organ, Chaos and ACDC datasets, showing that this approach achieves competitive performance over other segmentation methods, with fewer parameters and lower FLOPs.

In augmented reality, for achieving geometric consistency in the perspective projection virtual-real, we propose a semantic consistency method to achieve the fusion between virtual and real objects with selected segmented objects in the real scene as references. The proposed framework maintains the three-dimensional structure of the scene by satisfying the global semantic map of the real scene. It takes the segmented objects in the scene as the basic unit, and executes the virtual and real fusion for ensuring the accuracy of the relative geometric position of the virtual objects. In addition, a multi-task network architecture is proposed to optimize the camera parameters based on the scene segmentation. The experiment results demonstrate the effectiveness of the proposed augmented reality geometric consistency framework, and confirm that our strategy has the capability of fusing the virtual and real geometric consistency.

A proper outfit should consist of different categories of items that are visually compatible and share a similar style. Besides, personal aesthetic preference is also an important criterion when creating an overall outfit. Nevertheless, only few studies deal with aesthetic information in previous work on outfit composition. In this paper, we exploit both sequential learning and deep aesthetic network to train an end-to-end model for composing aesthetic outfits automatically. In detail, we firstly introduce a bidirectional long short-term memory (Bi-LSTM) model to discover the concept of compatibility among fashion items in an outfit. Then, an aesthetic-based model is proposed to parallel supervise the Bi-LSTM model so that the trained model can capture the aesthetic features from outfits. Meanwhile, we leverage visual-semantic descriptions to guarantee the uniqueness of each fashion item in an outfit. Moreover, to evaluate the effectiveness and practicability of the proposed model, we also design two representative tasks. One is evaluating the aesthetic scores of existing outfits. The other is composing aesthetic outfits conditioned on the given fashion item. Considering that aesthetic is highly correlated with personal preference, we additionally conduct the experiment that we can create proper outfits according to personal aesthetic preferences. Extensive experiments indicate that our method can generate aesthetic outfits that meet personal preferences and outperform the state-of-the-art.

Regarding depth and camera pose estimation tasks on the image sequence, we propose a monocular visual odometry frame named VONN on basis of convolutional neural network and recurrent neural network in this paper. The entire network frame is composed of two parts, i.e. depth estimator and camera 6-DoF position estimator. Besides, the paper also proposes a geometric consistency loss on this basis to train the network. By comparing VONN with other approaches in the accuracy of 6-DoF position estimation and accuracy of depth estimation, our method achieves better results and verifies the effectiveness of the frame and 3D geometric consistency loss proposed in this paper.

Abstract Generating realistic spray details in liquid simulations remains computationally expensive. This paper proposes a data‐driven method to simulate high‐resolution sprays on low‐resolution grids by retrieving details with the most compatible details from a precomputed repository efficiently. We first employ a random forest‐based distance (RFD) to measure the similarity of liquid regions. In consideration of spatiotemporal relationships between one liquid region and its neighbors, we define a multinary label for RFD instead of the original binary one. Our improved RFD enables us to retrieve details that fit ground truth the best. To ensure temporal continuity of our result and to generate new details from existing ones, we formulate a series of forests with a training set from different time steps. Then, we synthesize results of each forest according to their distances. Finally, we put the synthesis result in correct positions to generate desired sprays motion. In our method, a state‐of‐the‐art cascade forest is employed for a higher accuracy. Several experiments with various grid resolutions validate our method both in visual effect and computational cost.

ABSTRACT This paper puts forward a hierarchical method of fluid surface modeling in natural landscapes. The proposed method produces a visually plausible surface geometry with the texture from a single video image recorded by a standard video device. In contrast with the conventional physically based fluid simulation, our method computes preliminary results using empirical method and adopts Stokes wave model to obtain the reconstruction result. We illustrate the working of system with a wide range of possible scene, and a qualitative evaluation of our method is provided to verify the quality of the surface geometry. The experiment shows that the method can meet the requirement of real‐time performance and the reality of the fluid. Copyright © 2013 John Wiley & Sons, Ltd.