This paper introduces CrossDR, a novel approach unifying 2D and 3D feature representations through a context-aware cross-attention mechanism for diabetic retinopathy (DR) classification in 3D Optical Coherence Tomography (OCT) scans. The model uncovers critical diagnostic cues embedded in 3D-OCT images, which offer rich information for DR detection. The framework comprises three core components: the Lightweight Attention (LA) block, a CNN-based encoder, and the Context-Aware Cross-Attention (CA)2block. The LA block highlights volume-specific features that are crucial for DR diagnosis. It achieves this by applying a series of 1×1 convolutions followed by softmax operations, enabling the extraction of spatially informative DR-specific features. Meanwhile, the CNN-based encoder extracts high-level semantic features from the 3D-OCT slices, offering a robust representation of retinal structures. The (CA)2block further boosts the model’s performance by dynamically capturing and reweighting feature dependencies. This block models relationships between feature maps at different abstraction levels, improving the discriminative power of the extracted features. By integrating these refined features through the (CA)2block, the model achieves accurate DR classification. The CrossDR framework was evaluated on 481 volumetric OCT images, outperforming existing state-of-the-art methods with an accuracy of 94%. Multiple experiments and ablation studies were conducted to assess our model’s performance.

This study introduces an innovative framework designed specifically for accurate colon segmentation in abdomen CT scans, tackling the distinct challenges inherent to this task. Building upon well-established 2D segmentation models, our architecture adeptly incorporates 3D contextual information via a novel method that generates an attention map for a given slice by considering its neighboring slices in a sequence. Our approach accomplishes effective colon segmentation without requiring complex 3D convolutional neural networks (CNNs) or Long Short-Term Memory (LSTM) networks by combining 2D CNNs. Validated on a dataset of 98 CT scans from 49 patients, the architecture exhibits notable performance, successfully capturing nuanced details crucial for precise colon segmentation. The experiments encompass a thorough examination of model selection and cross-validation, providing valuable insights into the efficacy of our proposed approach. The outcomes underscore the potential for streamlined colon segmentation in medical imaging by judiciously integrating 2D and 3D information, employing solely 2D networks, and mitigating challenges associated with 3D networks. The code for model architecture is available at: https://github.com/Samir-Farag/ICIP2024.git

Early diagnosis of colorectal polyps, before they turn into cancer, is one of the main keys to treatment. In this work, we propose a framework to help radiologists in reading CT scans and identifying candidate CT slices that have polyps. We propose a colorectal polyps detection approach which consists of two cascaded stages. In the first stage, a CNN-based model is trained and validated to detect polyps in axial CT slices. To narrow down the effective receptive field of the detector neurons, the colon regions are segmented and then fed into the network instead of the original CT slice. This drastically improves the detection and localization results, e.g., the mAP is increased by 36%. To reduce the false positives generated by the detector, in the second stage, we propose a multi-view network (MVN) that classifies polyp candidates. The proposed MVN classifier is trained using sagittal and coronal views corresponding to the detected axial views. The approach is tested in 50 CTC-annotated cases, and the experimental results confirm that after the classification stage, polyps can be detected with an AUC $\sim 95.27 \%$.

Colon cancer, also known as colorectal cancer, is a significant health concern, with increasing incidence rates, particularly among individuals under 50. This rise has led experts to recommend the introduction of regular screenings at 45 years of age for adults at average risk. Early detection through such screenings can identify precancerous polyps, allowing their removal before they develop into cancer. This proactive approach has the potential to reduce colorectal cancer deaths by up to 60%. In addition, research indicates that people diagnosed before age 50 have better survival rates, which emphasizes the importance of early diagnosis. Therefore, adhering to recommended screening guidelines and promptly addressing concerns can significantly improve outcomes and save lives. This study proposes a framework to detect and localize colon cancer indicators (polyps) in 3D medical images, which has been applied to help radiologists read computed tomography (CT) scans and identify candidate CT slices with colonic polyps. This work’s main benefit is introducing AI-based colon abnormalities detection framework that the radiologist could miss by using different approaches and modalities. In this work, we propose an automatic detection approach for colorectal polyps consisting of two cascade stages. In the first stage, a Convolutional Neural Network (CNN) model is trained to detect polyps in axial CT slices; a CNN model has been fed by the segmented colon wall CT slices instead of the original CT slices. Using segmented images as input to the CNN model has drastically improved detection and localization results; for example, the mean average precision (mAP) for detection increases by 36%. To reduce false positives generated by the detector, the second stage classifier is deployed to exploit the different views of CT scans instead of the axial view only. So, the classifier is trained using the 2D images of axial views, i.e., the candidate polyps generated by the detector, as well as their corresponding 2D images of sagittal and coronal views. Different experiments were conducted to evaluate the proposed work, including the Fly-In approach to visualize and detect polyps within the 3D colon model, which was successfully used to train a CNN-based model that detects polyps with mAP∼ 97.1%. The second approach is to use the axial CT view after removing other organs except the colon and using A.I model to detect polyps within the 2D axial view with mAP∼ 86%. Finally, the MVN classification approach successfully identified polyps after the classification stage with an area

In this paper, we propose a novel framework that measures the engagement level of students either in a class environment or in an e-learning environment. The proposed framework captures the user's video and tracks their faces' through the video's frames. Different features are extracted from the user's face e.g., facial fiducial points, head pose, eye gaze, learned features, etc. These features are then used to detect the Facial Action Coding System (FACS), which decomposes facial expressions in terms of the fundamental actions of individual muscles or groups of muscles (i.e., action units). The decoded action units (AUs) are then used to measures the student's willingness to participate in the learning process (i.e., behavioral engagement) and his/her emotional attitude towards learning (i.e., emotional engagement). This framework will allow the lecturer to receive a real-time feedback from facial features, gaze, and other body kinesics. The framework is robust and can be utilized in numerous applications including but not limited to the monitoring the progress of students with various degrees of learning disabilities, and the analysis of nerve palsy and its effects on facial expression and social interactions.

Intra-oral cameras are essentially enabling dentists to capture images of difficult-to-reach areas in the mouth. Oral dental applications based on visual data pose various challenges such as low lighting conditions and saliva. We introduce an approach to stitch images of human teeth that are captured by an intra-oral camera. In such monocular image matching, a low rate of features on teeth surfaces causes a problem leading to a mismatch between teeth images. In this paper, we propose an approach to improve the matching in these low-texture regions. First, normals of tooth surface is extracted using a shape from shading. Due to the oral environment, the surface normals impact many of imprecise values; hence we formulate an algorithm to rectify these values and generate normal maps. The normal maps reveals the impacted geometric properties of the images inside an area, boundary, and shape. Second, the normal maps are used to detect, extract and match the corresponding features. Finally, to enhance the stitching process for these unidealized data, normal maps are used to estimate as-projective-as-possible warps. The proposed approach outperforms the state-of-the-art auto-stitching approach and shows a better performance in such cases of low-texture regions.

In this paper, a new visualization method for tubular inner surfaces is proposed, denoted by “Fly-In”. The approach uses a virtual camera that moves along the inner surface's centerline, obtaining projections of the surrounding view, formed of small 3D topological rings, within the tube that is rendered as a 2D rectangular image. A new visualization loss measure is also presented, which utilizes the projection direction of the camera axis, the surface normal and the ratio of the camera focal length and the surface distance. A comparative study of Fly-in versus two state-of-the-art methods (Fly-Through and Fly-Over) is conducted on 3D reconstructed colons from CT Colonography (CTC) patient data. Quantitative results are computed using the new loss measure and visualized using color coding. Results show high performance for the proposed Fly-In approach with respect to the state-of-art.

A set of facial Action Units (AUs) is activated due to the movement of facial muscles in response to a person's internal emotion. The activated facial action units appear around sparse regions on the face e.g., the mouth and eyes. This group occurrence of AUs reveals the semantic relationships among them. These relationships should be considered in the detection process of AUs. Therefore, we propose an approach that learns these semantic relationships using a multi-label deep learning architecture. The sparse patches of the AUs are used in the proposed approach instead of the whole facial region. To handle these sparse patches, we propose a region-based network instead of the well-known convolutional network and the locally connected network. Moreover, unlike the current approaches, which define facial regions using a uniform grid, the proposed region-based architecture, define patches around facial landmarks. This overcomes the region displacement problem of the uniform grid. Usually, the AUs classification suffers from the high skewness factor problem. To overcome this problem, we use a weighted loss function. We conduct our experiments on two standard benchmarks: BP4D and FERA17. f1score, the area under the ROC (AROC) and the area under the precision-recall curve (APR) are used as performance measures. Compared to the standard convolutional layer, the proposed patch-based layer is more effective in capturing the required structural features of the face and learns the correlations among AUs under pose variations. Also, it has been shown that the proposed approach outperforms the state-of-the-art methods.

Student behavior and lecturer oversight in the classroom is known to modulate study behaviors and impact performance and learning outcomes, but cannot at present be managed for distance learning students. Quantifying and automatically measuring student engagement during lectures in a scalable and accessible manner for these students is essential for improving academic success, but has not been studied widely in natural distance learning environments. We collect video recordings from a screen-mounted camera of students studying online lectures in a mostly unstructured setting and gather annotations from a panel of humans for assessing student engagement levels. We present results on the prediction of different representations of engagement, both with subject-independent and individual-specific models, and quantify the performance gap between the generalized and personalized models for engagement prediction. While the subject-independent performance is challenged by data sparsity, results show that the individual-specific models can predict engagement well even with very few labeled examples.

In this study, the authors learn a similarity measure that discriminates between inter‐class and intra‐class samples based on a statistical inference perspective. A non‐linear combination of Mahalanobis is proposed to reflect the properties of a likelihood ratio test. Since an object's appearance is influenced by the identity of the object and variations in the capturing process, the authors represent the feature vector, which is the difference between two samples in the differences space, as a sample that is drawn from a mixture of many distributions. This mixture consists of the identities distribution and other distributions of the variations in the capturing process, in case of dissimilar samples. However, in the case of similar samples, the mixture consists of the variations in the capturing process distributions only. Using this representation, the proposed similarity measure accurately discriminates between inter‐class and intra‐class samples. To highlight the good performance of the proposed similarity measure, it is tested on different computer vision applications: face verification and person re‐identification. To illustrate how the proposed learning method can easily be used on large scale datasets, experiments are conducted on different challenging datasets: labelled faces in the wild (LFW), public figures face database, ETHZ and VIPeR. Moreover, in these experiments, the authors evaluate different stages, for example, features detector, descriptor type and descriptor dimension, which constitute the face verification pipeline. The experimental results confirm that the learning method outperforms the state‐of‐the‐art.

Face recognition in the wild can be defined as recognizing individuals unabated by pose, illumination, expression, and uncertainties from the image acquisition. In this paper, we propose a framework recognizing human faces under such uncertainties by focusing on the pose problem while considering the other factors together. The proposed work introduces an automatic front-end stereo-based system, which starts with image acquisition and ends by face recognition. Once an individual is detected by one of the stereo cameras, its facial features are identified using a facial features extraction model. These features are used to steer the second camera to see the same subject. Then, a stereo pair is captured and 3D face is reconstructed. The proposed stereo matching approach carefully handles illumination variance, occlusion, and disparity discontinuity. The reconstructed 3D shape is used to synthesize virtual 2D views in novel poses. All these steps are done off-line in an Enrollment stage. To recognize a face from a 2D image, which is captured under unknown environmental conditions, another fast on-line stage starts by facial features detection. Then, a facial signature is extracted from patches around these facial features. Finally, this probe image is matched against the closest synthesized images. Experiments are conducted on different public databases from where we investigate the effect of each component of the proposed framework on the recognition performance. The results confirm that without training and with automatic features extraction, our proposed face recognition at a distance approach outperforms most of the state-of-the-art approaches.

Bone mineral density (BMD) measurements and fracture analysis of the spine bones are restricted to the Vertebral bodies (VBs), especially the trabecular bones (TBs). In this paper, we propose a novel, fast, and robust 3D framework to segment VBs and trabecular bones in clinical computed tomography (CT) images without any user intervention. The Matched filter is employed to detect the VB region automatically. To segment the whole VB, the graph cuts method which integrates a linear combination of Gaussians (LCG) and Markov Gibbs Random Field (MGRF) is used. Then, the cortical and trabecular bones are segmented using local volume growing methods. Validity was analyzed using ground truths of data sets (expert segmentation) and the European Spine Phantom (ESP) as a known reference. Experiments on the data sets show that the proposed segmentation approach is more accurate than other known alternatives.

Osteoporosis is a bone disease characterized by a reduction in bone mass, resulting in an increased risk of fractures. To diagnose the osteoporosis accurately, bone mineral density (BMD) measurements and fracture analysis (FA) of the Vertebral bodies (VBs) are required. In this paper, we propose a robust and 3D shape based method to segment VBs in clinical computed tomography (CT) images in order to make BMD measurements and FA accurately. In this experiment, image appearance and shape information of VBs are used. In the training step, 3D shape information is obtained from a set of data sets. Then, we estimate the shape variations using a distance probabilistic model which approximates the marginal densities of the VB and background in the variability region. In the segmentation step, the Matched filter is used to detect the VB region automatically. We align the detected volume with 3D shape prior in order to be used in distance probabilistic model. Then, the graph cuts method which integrates the linear combination of Gaussians (LCG), Markov Gibbs Random Field (MGRF), and distance probabilistic model obtained from 3D shape prior is used.

In this paper, we propose a new 3D framework to identify and segment VBs and TBs in clinical computed tomography (CT) images without any user intervention. The Matched filter is employed to detect the VB region automatically on axial axis. To identify the VB on coronal and sagittal axis, we use a new developed approach based on 4 points automatically placed on cortical shell. To segment the identified VB, the graph cuts method which integrates a linear combination of Gaussians (LCG) and Markov Gibbs Random Field (MGRF) are used. Then, the cortical and trabecular bones are segmented using local volume growing methods. Experiments on the data sets show that the proposed segmentation approach is more accurate than other known alternatives.

Exposure levels (X-ray tube amperage and peak kilovoltage) are associated with various noise levels and radiation dose. When higher exposure levels are applied, the images have higher signal to noise ratio (SNR) in the CT images. However, the patient receives higher radiation dose in this case. In this paper, we use our robust 3D framework to segment vertebral bodies (VBs) in clinical computed tomography (CT) images with different noise levels. The Matched filter is employed to detect the VB region automatically. In the graph cuts method, a VB (object) and surrounding organs (background) are represented using a gray level distribution models which are approximated by a linear combination of Gaussians (LCG). Initial segmentation based on the LCG models is then iteratively refined by using Markov Gibbs random field(MGRF) with analytically estimated potentials. Experiments on the data sets show that the proposed segmentation approach is more accurate and robust than other known alternatives.

Bone mineral density (BMD) measurements and fracture analysis of the spine bones are restricted to the Vertebral bodies (VBs). In this paper, we propose a novel 3D shape based method to segment VBs in clinical computed tomography (CT) images without any user intervention. The proposed method depends on both image appearance and shape information. 3D shape information is obtained from a set of training data sets. Then, we estimate the shape variations using a distance probabilistic model which approximates the marginal densities of the VB and background in the variability region. To segment a VB, the Matched filter is used to detect the VB region automatically. We align the detected volume with 3D shape prior in order to be used in distance probabilistic model. Then, the graph cuts method which integrates the linear combination of Gaussians (LCG), Markov Gibbs Random Field (MGRF), and distance probabilistic model obtained from 3D shape prior is used. Experiments on the data sets show that the proposed segmentation approach is more accurate than other known alternatives.

This paper introduces a framework for long-distance face recognition using dense and sparse stereo reconstruction, with texture of the facial region. Two methods to determine correspondences of the stereo pair are used in this paper: (a) dense global stereo-matching using maximum-a-posteriori Markov Random Fields (MAP-MRF) algorithms and (b) Active Appearance Model (AAM) fitting of both images of the stereo pair and using the fitted AAM mesh as the sparse correspondences. Experiments are performed using combinations of different features extracted from the dense and sparse reconstructions, as well as facial texture. The cumulative rank curves (CMC), which are generated using the proposed framework, confirms the feasibility of the proposed work for long distance recognition of human faces.

Parametric density estimation is widely used to solve many image processing problems. We examined the parametric estimation using linear combination of 1D Gaussians in many works. In this work, we extend our model to estimate density of the colors in color images. We approximate the marginal density of each class in the empirical probability density function by a 3D Gaussian distribution. Then, the deviation between the estimated and the empirical densities is modelled using a linear combination of 3D Gaussians with positive and negative components. We estimate the parameters of this model using our modified EM algorithm. The proposed framework demonstrates very promising experimental results of color images labelling and can be integrated with many other frameworks.

We present a 3D segmentation technique of trabecular (cancellous) bones in CT images of Vertebral bodies (VBs). In order to be used for Bone Mineral Density (BMD) measurements, the cortical and trabecular bones are subsequently segmented using graph cuts method and local volume growing methods separately. In the final step, we measure our segmentation accuracy for each method. Validity was analyzed using ground truths of data sets and the European Spine Phantom (ESP). Preliminary results are very encouraging and a reproducibility of the results was achieved for 16 data sets. The average segmentation error is below 2.0% for both methods.

We introduce a framework for face recognition at a distance based on sparse-stereo reconstruction. We develop a 3D acquisition system that consists of two CCD stereo cameras mounted on pan-tilt units with adjustable baseline. We first detect the facial region and extract its landmark points, which are used to initialize an AAM mesh fitting algorithm. The fitted mesh vertices provide point correspondences between the left and right images of a stereo pair; stereo-based reconstruction is then used to infer the 3D information of the mesh vertices. We perform experiments regarding the use of different features extracted from these vertices for face recognition. The cumulative rank curves (CMC), which are generated using the proposed framework, confirms the feasibility of the proposed work for long distance recognition of human faces with respect to the state-of-the-art [3].

This work proposes a new MAP-based segmentation framework of multimodal images. In this work a joint MGRF model is used to describe the image. The main focus here is a more accurate model identification. For a known number of classes in the given image, the empirical distributions of this image signals are precisely approximated by a LCG distributions with positive and negative components. Gibbs potential, which is used to identify the spatial interaction between the neighboring pixels, is analytically estimated. Finally, an energy function using the previous models is formulated and is globally minimized using graph cuts. Experiments show that the developed technique gives promising accurate results compared to other known algorithms.

We propose a new approach that approximates an empirical probability density function of scalar data with a linear combination of Gaussians (LCG). The proposed algorithm approximates the marginal density of each class using a Gaussian distribution. Number of the classes and their distributions parameters are estimated using a new Akaike Information Criterion (AlC)-type criterion and the Expectation- Maximization (EM) approach. Each class does not follow perfect Gaussian distribution so we refine the initial LCG model using a modified EM algorithm. The modified EM algorithm approximates the marginal density of each class using a LCG with positive and negative components. Experiments in segmenting multimodal medical images show that the developed technique gives promising accurate results.

This paper proposes a new analytical method for estimating parameters of a homogeneous isotropic Potts model with an asymmetric Gibbs potential function. The model is generalized by including both pairwise and triple cliques. The maximum likelihood estimates of the cliques potentials are obtained by a further elaboration of the approximate analytical estimator proposed in. Experiments with synthetic textures have shown that our potential estimates are more accurate and practicable than their counterparts obtained with classical methods.