Surgery video summarization can help medical professionals quickly gain the insight into the surgical process for the surgical education and skill evaluation. However, existing methods are unable to efficiently summarize information to satisfy medical professionals. Since it is challenging to summarize the video while balancing the information richness and diversity. In this paper, we propose a dynamic surgery video summarization framework (DSVS). We first used a multitask learning network to perceive and comprehend surgical action triplet components and phases. An information contribution module then measures the frame-level importance using the predicted triplets. A two-stage strategy which involves phase recognition and change-point detection further applied to divide each phase of the surgical videos into shots. Finally, A multi-objective zero-one programming model is formulated to select the optimal subset of shots by simultaneously maximizing intra-shot information contribution and minimizing inter-shot information similarity. Experimental results on two surgical video datasets show the framework can generate summaries that encompass crucial and diverse content. Clinical validations indicate the framework is capable of summarizing the information expected by surgeons. The source code can be found at https://github.com/syypretend/DSVS.

Negative emotional states, such as anxiety and depression, pose significant challenges in contemporary society, often stemming from the stress encountered in daily activities. Stress (state or level) recognition is a crucial prerequisite for effective stress management and intervention. Presently, wearable devices have been employed to capture physiological signals and analyze stress states. However, their constant skin contact can lead to discomfort and disturbance during prolonged monitoring. In this paper, a peak attention-based multitasking framework is presented for non-contact stress recognition. The framework extracts rPPG signals from RGB facial videos, utilizing them as inputs for a novel multi-task attentional convolutional neural network for stress recognition (MTASR). It incorporates peak detection and HR estimation as auxiliary tasks to facilitate stress recognition. By leveraging multi-task learning, MTASR can utilize information related to stress physiological responses, thereby enhancing feature extraction efficiency. For stress recognition, two binary classification tasks are applied: stress state recognition and stress level recognition. The model is validated on the UBFC-Phys public dataset and demonstrates an accuracy of 94.33% for stress state recognition and 83.83% for stress level recognition. The proposed method outperforms the dataset's baseline methods and other competing approaches.

Medical professionals rely on surgical video retrieval to discover relevant content within large numbers of videos for surgical education and knowledge transfer. However, the existing retrieval techniques often fail to obtain user-expected results since they ignore valuable semantics in surgical videos. The incorporation of rich semantics into video retrieval is challenging in terms of the hierarchical relationship modeling and coordination between coarse- and fine-grained semantics. To address these issues, this paper proposes a novel semantic-preserving surgical video retrieval (SPSVR) framework, which incorporates surgical phase and behavior semantics using a dual-level hashing module to capture their hierarchical relationship. This module preserves the semantics in binary hash codes by transforming the phase and behavior similarities into high- and low-level similarities in a shared Hamming space. The binary codes are optimized by performing a reconstruction task, a high-level similarity preservation task, and a low-level similarity preservation task, using a coordinated optimization strategy for efficient learning. A self-supervised learning scheme is adopted to capture behavior semantics from video clips so that the indexing of behaviors is unencumbered by fine-grained annotation and recognition. Experiments on four surgical video datasets for two different disciplines demonstrate the robust performance of the proposed framework. In addition, the results of the clinical validation experiments indicate the ability of the proposed method to retrieve the results expected by surgeons. The code can be found at https://github.com/trigger26/SPSVR.

Facial video-based remote physiological measurement aims to estimate remote photoplethysmography (rPPG) signals from human facial videos and then measure multiple vital signs (e.g., heart rate, respiration frequency) from rPPG signals. Recent approaches achieve it by training deep neural networks, which normally require abundant facial videos and synchronously recorded photoplethysmography (PPG) signals for supervision. However, the collection of these annotated corpora is not easy in practice. In this paper, we introduce a novel frequency-inspired self-supervised framework that learns to estimate rPPG signals from facial videos without the need of ground truth PPG signals. Given a video sample, we first augment it into multiple positive/negative samples which contain similar/dissimilar signal frequencies to the original one. Specifically, positive samples are generated using spatial augmentation; negative samples are generated via a learnable frequency augmentation module, which performs non-linear signal frequency transformation on the input without excessively changing its visual appearance. Next, we introduce a local rPPG expert aggregation module to estimate rPPG signals from augmented samples. It encodes complementary pulsation information from different face regions and aggregates them into one rPPG prediction. Finally, we propose a series of frequency-inspired losses, i.e., frequency contrastive loss, frequency ratio consistency loss, and cross-video frequency agreement loss, for the optimization of estimated rPPG signals from multiple augmented video samples. We conduct rPPG-based heart rate, heart rate variability, and respiration frequency estimation on five standard benchmarks. The experimental results demonstrate that our method improves the state of the art by a large margin.

Depression is a mental disorder that causes pain to people and society and is also the largest cause of disability in the world. Intelligent early screening of depression is of great benefit for patients to obtain better diagnoses and treatment. However, previous low-precision detection methods based on facial vision heavily rely on computing resources, which hinders the wide application of automatic depression diagnoses. Therefore, this article proposes an intelligent method for multiscene automatic depression symptom detection, which uses an efficient and convenient cloud-edge collaboration framework combined with negative emotion monitoring and cross-scale facial feature analysis. We deploy a shallow model (EdgeER) on the edge server and a deep model (C-DepressNet) on the cloud server. EdgeER is used to quickly detect negative user emotions and screen user data. C-DepressNet is used to analyze degrees of depression with high precision. The experimental results show that our cloud-edge collaboration framework has superior performance in depression detection accuracy and service response times.

With the application of the Internet of Medical Things (IoMT) in minimally invasive surgery (MIS), surgeons now have a better chance at hard-to-treat cases by carrying out more complicated MIS workflows. However, a scheduled surgical workflow is often required to be updated based on the patient’s internal tissue states. Perioperative complications could occur if in-time adjustments are lacking in the operating rooms when needed. To help manage the uncertainty of live surgical workflows in the IoMT environment, we propose a MIS safety improvement framework. It helps surgeons in predicting surgical workflows with limited MIS video frames by embedding our proposed model GuidedNet. To predict future surgical activities, we first build three isomorphic neural networks to capture the spatiotemporal information. Then, we establish a guidance fusion module to handle the contextual information. It guides the GuidedNet to recognize the surgical stage. Moreover, we build a novel joint loss function to train the GuidedNet to predict the future surgical stage. We evaluate the approach on a large data set that contains 80 cholecystectomy videos (Cholec-80) and compare it with the state of the art. Experiments show that the GuidedNet can assist surgeons in carrying out MIS as well as guide the next stage of surgery for improving surgical safety. Comparing to the state of the art, our approach can obtain better predict accuracy (up to 79%) with less computing resource consumption. The result also shows that our approach has a high application prospect in video classification in other Internet of Things scenarios.

Negative mood states include tension, depression, anger, fatigue, and confusion, which represent the weak internal emotions of a human. Negative mood states exert adverse impact on individuals' ability to make rational decisions, which entails the practicable method of negative mood state detection. The most commonly used negative mood state detection methods are based on the psychological scale, which requires additional work and brings inconvenience to the subject in the application scenarios. To overcome this challenge, this paper proposes a novel non-contact negative mood state detection method according to the knowledge of affective computing. The POMS-net model is used to extract temporal-spatial features from visible and infrared thermal videos, and the negative mood state detection is realized using data reliability-focused multi-modal fusion. The proposed method is verified using the HDT-BR dataset collected in the aerospace medicine experiment "Earth-Star II" and the VIRI public dataset. The experimental results on the datasets verify that our method outperforms the comparison methods.

Gastroscopy is a widely adopted method for locating gastric lesions and performing the early screening and diagnosis of gastric cancer (GC). However, the effectiveness of traditional GC screening methods depends on the medical skills of the gastroscopy specialist. A lack of knowledge and experience may lead to misdiagnosis and mistreatment, especially in small-scale hospitals. Recently, there has been a significant increase in studies on data-driven computer-aided diagnosis techniques. In this article, we propose a novel intelligent decision-making method for GC screening (ID-GCS), a multimodal semantic fusion-based data-driven decision-making system. ID-GCS exploits a hybrid attention mechanism to extract textual semantics from multimodal gastroscopy reports and performs semantic fusion to integrate the semantics of textual gastroscopy reports and images, resulting in improved interpretability of gastroscopy findings. We evaluated ID-GCS using a real gastroscopy report dataset, and experimental results show that compared with state-of-the-art methods, ID-GCS achieves better sensitivity and accuracy in GC screening.

Abstract Compared to patients readmitted to general wards, readmitted patients in the intensive care unit (ICU) are exposed to higher mortality rates and prolonged hospital stays. Moreover, the readmission of ICU patients brings pressing challenges for ICU management. Most models are devoted to identifying the risk factors and developing classification models that can predict whether ICU patients will be readmitted. Though these models are prominent, they do not provide estimates for the frequency of readmissions. This paper establishes a prediction model, hybrid feature selection‐LightGBM (HFS‐LightGBM), to evaluate the probability and frequency of ICU patient readmissions empirically. In terms of feature selection, a hybrid feature selection (HFS) algorithm for LightGBM combines the filter and wrapper methods. Pearson's correlation coefficient is employed in the filter procedure. Then we adopt the targeted LightGBM classifier along with the recursive feature elimination and cross‐validated (RFECV) to produce the optimal feature subset. Additionally, the hyperparameters of the HFS‐LightGBM are optimized. The HFS‐LightGBM is employed on the real‐world ICU dataset containing 1722 patients' electronic health records. This model outperforms the current prevailing readmission models. The identified frequency can assist doctors in making specific interventions for patients to reduce the ICU readmission rate.

Three-dimensional display and virtual reality technology have been applied in minimally invasive surgery to provide doctors with a more immersive surgical experience. One of the most popular systems based on this technology is the Da Vinci surgical robot system. The key to build the in vivo 3-D virtual reality model with a monocular endoscope is an accurate estimation of depth and motion. In this article, a fully unsupervised learning method for depth and motion estimation using the continuous monocular endoscopic video is proposed. After the detection of highlighted regions, EndoMotionNet and EndoDepthNet are designed to estimate ego-motion and depth, respectively. The timing information between consecutive frames is considered with a long short-term memory layer by EndoMotionNet to enhance the accuracy of ego-motion estimation. The estimated depth value of the previous frame is used to estimate the depth of the next frame by EndoDepthNet with a multimode fusion mechanism. The custom loss function is defined to improve the robustness and accuracy of the proposed unsupervised-learning-based method. Experiments with the public datasets verify that the proposed unsupervised-learning-based continuous depth and motion estimation method can effectively improve the accuracy of depth and motion estimation, especially after processing the frame.

Upper gastrointestinal (UGI) cancer has been identified as one of the ten most common causes of cancer deaths globally. UGI cancer screening is critical to improving the survival rate of UGI cancer patients. While many approaches to UGI cancer screening rely on single-modality data such as gastroscope imaging, limited studies have been dedicated to UGI cancer screening exploiting multisource and multimodal medical data, which could potentially lead to improved screening results. In this paper, we propose semantic-level cancer-screening network (SCNET), a framework for UGI cancer screening based on semantic-level multimodal upper gastrointestinal data fusion. Specifically, the proposed SCNET consists of a gastrointestinal image recognition flow and a textual medical record processing flow. High-level features of upper gastrointestinal data are extracted by identifying effective feature channels according to the correlation between the textual features and the spatial structure of the image features. The final screening results are obtained after the data fusion step. The experimental results show that the improvement of our approach over the state-of-the-art ones reached 4.01% in average. The source code of SCNET is available at https://github.com/netflymachine/SCNET.

Automatic acetowhite lesion segmentation in colposcopy images (cervigrams) is essential in assisting gynecologists for the diagnosis of cervical intraepithelial neoplasia grades and cervical cancer. It can also help gynecologists determine the correct lesion areas for further pathological examination. Existing computer-aided diagnosis algorithms show poor segmentation performance because of specular reflections, insufficient training data and the inability to focus on semantically meaningful lesion parts. In this paper, a novel computer-aided diagnosis algorithm is proposed to segment acetowhite lesions in cervigrams automatically. To reduce the interference of specularities on segmentation performance, a specular reflection removal mechanism is presented to detect and inpaint these areas with precision. Moreover, we design a cervigram image classification network to classify pathology results and generate lesion attention maps, which are subsequently leveraged to guide a more accurate lesion segmentation task by the proposed lesion-aware convolutional neural network. We conducted comprehensive experiments to evaluate the proposed approaches on 3045 clinical cervigrams. Our results show that our method outperforms state-of-the-art approaches and achieves better Dice similarity coefficient and Hausdorff Distance values in acetowhite legion segmentation.

Recent studies have shown that there exhibits significantly imbalanced medical resource allocation across public hospitals. Patients, regardless of their diseases, tend to choose hospitals and physicians with a better reputation, which often overloads major hospitals while leaving others underutilized. Guiding patients to hospitals that can serve their treatment needs both timely and with good quality can make the best use of precious medical resources. Unfortunately, it remains one of the major challenges both for research and in practice. In this article, we propose a novel diversity-enhanced hierarchical physician recommendation approach to address this issue. We adopt matrix factorization to estimate physician competency and exploit implicit similarity relationships to improve the competency estimation of physicians that we are of little information of. We then balance the patient preference and physician diversity using two novel heuristic algorithms. We evaluate our proposed approach and compare it with the state of the art. Experiments show that our approach significantly improves both accuracy and recommendation diversity over existing approaches.

Time-series medical images are an important type of medical data that contain rich temporal and spatial information. As a state-of-the-art, computer-aided diagnosis (CAD) algorithms are usually used on these image sequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medical images to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, the existing CAD algorithms support analysis on each encrypted image but not on the whole encrypted image sequences, which leads to the loss of important temporal information among frames. To meet this challenge, a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time-series medical images encrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks are constructed to extract discriminative spatial features, and LSTM-based sequence analysis layers (HE-LSTM) are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weighted unit and a sequence voting layer are designed to incorporate both spatial and temporal features with different weights to improve performance while reducing the missed diagnosis rate. The experimental results on two challenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence that our framework can encode visual representations and sequential dynamics from encrypted medical image sequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constituting a significant margin of statistical improvement compared with several competing methods.

Abstract Gastroscopy is important for finding suspicious stomach lesions, screening for gastric cancer, and providing early diagnoses. Due to the differences in the levels of diagnosis and treatment among gastroscope doctors, clinical diagnosis based on gastroscopy is limited by low diagnostic sensitivity and specificity to gastric cancer. An assistive system for gastroscopy report analysis can be helpful to improve the success rate of gastric cancer detection. In this study, a homogeneous ensemble decision support system for gastric cancer screening (Endo‐GCS) that performs word segmentation, feature extraction, and gastric cancer screening on text‐based gastroscopy reports is proposed. The proposed Endo‐GCS method establishes a progressive local weighting algorithm that improves the overall prediction performance of the homogeneous ensemble model in gastric cancer screening. An optimal threshold estimation algorithm is developed to minimize the negative impact of misdiagnosis and missed diagnoses. Through a comparative experimental study using real gastroscopy report data, the pathological examination conclusion is the gold standard. The sensitivity of the proposed Endo‐GCS method is 88.27%, the specificity is 77.84%, and the accuracy is 82.11%, which significantly improved the sensitivity 65.49% and the accuracy 80.5% of the gastroscopic diagnosis results, respectively.

Abstract Endoscopy is an important tool for gastric cancer screening. Due to the lack of effective decision support system for endoscopy, the detection of gastric cancer in the clinic is usually with low sensitivity. In this paper, we propose a Genetic Algorithm optimized Neural Network (GAoNN) approach for gastric cancer detection based on endoscopy reports mining. Considering the fact that gastric cancer sensitivity can significantly improve the 5‐year survival rate of patients, both the prediction accuracy and the sensitivity are employed to construct a multiobjective optimization model for enhancing the classification performance of GAoNN. In particular, we extended an effective genetic algorithm Nondominated Sorting Genetic Algorithm II (NSGA‐II) to train a neural network and reduced the complexity in training hyperparameters and improved the efficiency by substituting the computationally intensive stochastic gradient descent (SGD) algorithm in a neural network. Specifically, we designed the novel crossover and mutation operators and modified the nondominated ranking and crowding distance sorting procedures in NSGA‐II for GAoNN. Through testing on 8,546 real‐world endoscopy reports, we show that GAoNN achieves a prediction accuracy up to 83.74%, which is better than several competitors by significantly increasing sensitivity to 83.14%. GAoNN also reduces the training time by 30.94% when compared with conventional SGD‐based training, which indicates the feasibility of GAoNN in clinical practice.

Cervical cancer ranks as the second most common cancer in women worldwide. In clinical practice, colposcopy is an indispensable part of screening for cervical intraepithelial neoplasia (CIN) grades and cervical cancer but exhibits high misdiagnosis rate. Existing computer-assisted algorithms for analyzing cervigram images have neglected that colposcopy is a sequential and multistate process, which is unsuitable for clinical applications. In this work, we construct a cervigram-based recurrent convolutional neural network (C-RCNN) to classify different CIN grades and cervical cancer. Convolutional neural networks are leveraged to extract spatial features. We develop a sequence-encoding module to encode discriminative temporal features and a multistate-aware convolutional layer to integrate features from different states of cervigram images. To train and evaluate the performance of C-RCNN, we leveraged a dataset of 4,753 real cervigrams and obtained 96.13% test accuracy with a specificity and sensitivity of 98.22% and 95.09%, respectively. Areas under each receiver operating characteristic curves are above 0.94, proving that visual representations and sequential dynamics can be jointly and effectively optimized in the training phase. Comparative analysis demonstrated the effectiveness of the proposed C-RCNN against competing methods, showing significant improvement over only focusing on a single frame. This architecture can be extended to other applications in medical image analysis.

With the fast advances in Internet technologies, social networks have become a major platform for social interaction, lifestyle demonstration, and message dissemination. Effective community detection in social networks helps to assess public sentiment, identify community leaders, and produce personalized recommendation. While different community detection approaches have been proposed in the literature, the trust model based detection schemes model user interactions as trust transfer, which helps to capture the implicit relation in the network. Unfortunately, trust model based detection schemes face acold startproblem, i.e., they cannot accurately model newly joined users as these users have few interactions for a duration after joining the network. In this paper, we propose TLCDA, a novel trust model based community detection algorithm. By enhancing the traditional trust computation with inter-node relation strength and similarity in social networks, TLCDA detects communities through coarse-grained K-Mediods clustering. Our evaluation on real social networks shows that the communities detected by TLCDA exhibit superior preference cohesion while satisfying the topology cohesion.

The essential role of the particular families of capacities and the capacity identification methods is to help the decision maker to deal with the exponential complexity inherent in the construction process of the capacity. The 2-additive capacities appear to be the most popular among the particular families of capacities since they permit to model interactions between criteria while preserving simplicity. Besides the preference with respect to the decision criteria, most of the capacity identification methods also need to provide the desired overall evaluations of the decision alternatives in the learning set, which is a time-consuming task for the decision maker. In this paper, we propose some models to identify 2-additive capacities only from a kind of refined preference information with respect to the decision criteria called the multicriteria correlation preference information (MCCPI). The MCCPI is a group of 2-D preference information which can be described and obtained by the refined diamond diagram. The common principle of the proposed models is to minimize the different kinds of deviations between the MCCPI and the most desired 2-additive capacity(ies). A multicriteria decision making example is presented to show the feasibility of the proposed methods, and a 2-D scale of the MCCPI is also introduced in the further discussion of the illustrative example.