Attribute-specific fashion retrieval aims to enhance fine-grained image retrieval by emphasizing the similarity of specific attributes. Current methods primarily rely on attention mechanisms to extract attribute-related visual features but face two key challenges: the limitations of coarse-grained localization in achieving fine-grained accuracy, and an imbalance between global and local perception, where excessive focus on local features can undermine overall performance. To address these issues, we propose the fashion microscope ProFashion, which achieves pixel-level attribute awareness through optimal transport and neural semantic aggregation. The framework begins by employing optimal transport to align semantic attributes with visual patterns from a global perspective, generating an attribute-visual value map that highlights distinctive regions while reducing interference. This is followed by simulating the human brain's perception of attribute feature patterns through superpixel generation and aggregation, capturing attribute-related features at the pixel semantic level and forming key semantic clusters that preserve microstructures. Building on this, an attribute graph is constructed to facilitate feature clustering, significantly enhancing the framework's capability to handle overlapping features and cross-scale relationships. Comprehensive experiments on the FashionAI, DeepFashion, and DARN datasets demonstrate the framework's effectiveness, achieving overall MAP improvements of 3.11%, 3.70%, and 3.49%, respectively. Additionally, the framework delivers relative average throughput gains of 26.94%, 22.22%, and 24.78% on the FashionAI, DeepFashion, and DARN datasets, respectively.

Due to the lack of physical touch, social distancing caused by COVID-19 makes it difficult to convey social intentions through handshakes. To mitigate this situation, one promising solution is to simulate distinguishable handshake patterns through haptic feedback devices during distant social communication. This paper reports a distant handshake scheme with multi-modal soft haptic gloves. To address the challenge of duplicating rich haptic stimuli of real handshakes in a compact glove, we extracted four haptic features including grip location, grip strength, skin temperature, and shaking frequency from abundant components of the haptic perception system to mimic handshake behaviors. To guide the interference-free spatial layout of multiple actuators in a limited hand-sized space, we measured the handshake contact area and grip strength of different handshake patterns, which together with the thermosensitivity of the human hand determine the grip location, grip strength, and salient thermal stimulating location. We developed a multi-modal soft haptic glove by rendering four features through pneumatic pressure, thermal, and vibrotactile stimuli, respectively. A user study was conducted to validate the performance of our glove, which set two states for each of the four features, showing over 90% accuracy in distinguishing 16 handshake patterns. Furthermore, a user study on the identification of social intentions yields the finding that distant handshakes with our haptic gloves can convey positive, neutral, and negative social intentions. These results inform the potential of distant handshakes with haptic gloves to convey social intentions in remote interactions including business, politics, and daily life in severe and post-pandemic situations, as well as in future metaverse-based society.

Despite the notable progress achieved by large-scale vision-language pre-training models in a wide range of multi-modal tasks, their performance often falls short in image-text matching challenges that require an in-depth understanding of structured representations. For instance, when distinguishing between texts or images that are generally similar but have distinct structured knowledge (such as entities and relationships in text, or objects and object attributes in images), the model's capabilities are limited. In this paper, we propose a advancing Multi-modal Structured Knowledge Representation with synergistic hard negative samples (MSKR), thereby significantly improving the model's matching capability for such data. Specifically, our model comprises a structured knowledge-enhanced encoder designed to bolster the structured knowledge inherent in textual data, such as entities, their attributes, and the relationships among these entities as well as structured knowledge within images, focusing on elements like objects and their attributes. To further refine the model's learning process, we produce both image and text challenging negative samples. Extensive experimental evaluations on the Winoground, InpaintCOCO, and MSCOCO benchmark reveal that MSKR significantly outperforms the baseline model, showcasing marked improvements 2.66% on average in structured representation learning compared to the baseline. Moreover, general representation results illustrate that our model not only excels in structured representation learning but also maintains its proficiency in general representation learning.

It is very difficult to directly detect objects on the entire large scale remote sensing image, due to the limited GPU memory. Moreover, there are no objects of interest in most areas of such a huge image, thus a lot of computational costs is wasted in dealing with these vain areas. Therefore, this paper proposes a Cropping Region Proposal Network (CRPN), which includes a weak semantic RPN for quickly locating interesting regions, and a dual-scale strategy for generating effective cropping regions. Cropping regions consist of small and large cropping scales for detecting various-scale objects including very small and very large objects, which is hard for existing methods. CRPN helps to detect effective regions of remote sensing image. Meanwhile, it is also modularized and can be easily connected with mainstream detectors to form an end-to-end detecting framework. Experiments on public DOTA dataset show that our CRPN is effective for filtering invalid regions to greatly reduce the computation burden, and helps to achieve more accurate object detection on large scale remote sensing images.

Segmentation of the left ventricle (LV) is essential for quantitative calculation of clinical indices for analyzing the cardiac contractile function. However, it is challenging to automatically segment small-contour cardiac magnetic resonance (CMR) images for traditional convolutional neural networks (ConvNets) because of their low robustness to scale variation. In this paper, we propose a multi-scale fusion learning method to advance the performance of ConvNets for the LV segmentation. To realize our multi-scale fusion learning, single-scale input and multi-scale output (SIMO) networks are firstly trained to construct a SIMO-based multi-scale fusion network (SIMO-based MSF_Net). The trained SIMO networks produce different-scale coarse prediction results which are then fused into another multi-scale network. Finally, the coarse results are progressively refined to yield finer segmentation results. Our multi-scale fusion learning is evaluated on MICCAI 2009 challenging database for the LV segmentation. Experimental results demonstrate the robustness of our SIMO-based MSF_Net for the segmentation of challenging CMR images and the metric of “Good contours” achieves 98.35% on the testing set, which is greatly improved compared with the state-of-the-art methods.

Haptic-based tissue stiffness perception is essential for palpation training system, which can provide the surgeon haptic cues for improving the diagnostic abilities. However, current haptic devices, such as Geomagic Touch, fail to provide immersive and natural haptic interaction in virtual surgery due to the inherent mechanical friction, inertia, limited workspace and flawed haptic feedback. To tackle this issue, we design a novel magnetic levitation haptic device based on electromagnetic principles to augment the tissue stiffness perception in virtual environment. Users can naturally interact with the virtual tissue by tracking the motion of magnetic stylus using stereoscopic vision so that they can accurately sense the stiffness by the magnetic stylus, which moves in the magnetic field generated by our device. We propose the idea that the effective magnetic field (EMF) is closely related to the coil attitude for the first time. To fully harness the magnetic field and flexibly generate the specific magnetic field for obtaining required haptic perception, we adopt probability clouds to describe the requirement of interactive applications and put forward an algorithm to calculate the best coil attitude. Moreover, we design a control interface circuit and present a self-adaptive fuzzy proportion integration differentiation (PID) algorithm to precisely control the coil current. We evaluate our haptic device via a series of quantitative experiments which show the high consistency of the experimental and simulated magnetic flux density, the high accuracy (0.28 mm) of real-time 3D positioning and tracking of the magnetic stylus, the low power consumption of the adjustable coil configuration, and the tissue stiffness perception accuracy improvement by 2.38 percent with the self-adaptive fuzzy PID algorithm. We conduct a user study with 22 participants, and the results suggest most of the users can clearly and immersively perceive different tissue stiffness and easily detect the tissue abnormality. Experimental results demonstrate that our magnetic levitation haptic device can provide accurate tissue stiffness perception augmentation with natural and immersive haptic interaction.

Left ventricle (LV) segmentation is crucial for quantitative analysis of the cardiac contractile function. In this paper, we propose a joint multi-scale convolutional neural network to fully automatically segment the LV. Our method adopts two kinds of multi-scale features of cardiac magnetic resonance (CMR) images, including multi-scale features directly extracted from CMR images with different scales and multi-scale features constructed by intermediate layers of standard CNN architecture. We take advantage of these two strategies and fuse their prediction results to produce more accurate segmentation results. Qualitative results demonstrate the effectiveness and robustness of our method, and quantitative evaluation indicates our method achieves LV segmentation with higher accuracy than state-of-the-art approaches.

Uterine fibroids segmentation in ultrasound images is of great importance in the definition of intra‐operative planning of ultrasound‐guided high‐intensity focused ultrasound (HIFU) therapy. However, it is challenging to obtain accurate, robust and efficient uterine fibroid segmentation due to low quality of ultrasound images. In this study, the authors propose a novel adaptive localised region and edge‐based active contour model using shape constraint and sub‐global information to accurately and efficiently segment the uterine fibroids in ultrasound images with robustness against initial contour. The authors first define adaptive local radius for the localised region‐based model and combine it with the edge‐based model to accurately and efficiently capture image's heterogeneous features and edge features. Then, they incorporate a shape constraint to reduce boundary leakage or excessive contraction to obtain more accurate segmentation. To overcome the initialisation sensitivity, they introduce the sub‐global information to prevent the curve from trapping into the local minima and obtain robust results. Furthermore, the authors optimise computation by adaptively sharing local region and employing the multi‐scale segmentation method to achieve efficient segmentation. The proposed method is validated by uterine fibroid ultrasound images in HIFU therapy and the results demonstrate that it can achieve accurate, robust and efficient segmentation.

In medical training especially in palpation surgery, it is important for surgeons to perceive tissue stiffness. We design a novel magnetic levitation haptic device based on electromagnetic principles to enhance the perception of tissue stiffness in a virtual environment. The user can directly sense virtual tissues by moving a magnetic stylus in the magnetic field generated by the coil array of our device. To fully use the effective magnetic field, we devise an adjustable coil array and provide a reasonable explanation for such design. Moreover, we design a control interface circuit and present a self-adaptive fuzzy proportion integration differentiation (PID) algorithm to precisely control the coil current. The quantitative experiment shows that the experimental and simulation data of our device are consistent and the proposed control algorithm contributes to increasing the accuracy of tissue stiffness perception. In qualitative experiment, we recruit 22 participants to distinguish tissues of different stiffness and detect tissue abnormality. The experimental results demonstrate that our magnetic levitation haptic device can provide accurate perception of tissue stiffness.