We present LoTUS, a novel Machine Unlearning (MU) method that eliminates the influence of training samples from pre-trained models, avoiding retraining from scratch. LoTUS smooths the prediction probabilities of the model up to an information-theoretic bound, mitigating its overconfidence stemming from data memorization. We evaluate LoTUS on Transformer and ResNet18 models against eight baselines across five public datasets. Beyond established MU benchmarks, we evaluate unlearning on ImageNet1k, a large-scale dataset, where retraining is impractical, simulating real-world conditions. Moreover, we introduce the novel Retrain-Free Jensen-Shannon Divergence (RF-JSD) metric to enable evaluation under real-world conditions. The experimental results show that LoTUS outperforms state-of-the-art methods in terms of both efficiency and effectiveness. Code: https://github.com/cspartalis/LoTUS.

MOTIVATION: Identifying antibody binding sites, is crucial for developing vaccines and therapeutic antibodies, processes that are time-consuming and costly. Accurate prediction of the paratope's binding site can speed up the development by improving our understanding of antibody-antigen interactions. RESULTS: We present ParaSurf, a deep learning model that significantly enhances paratope prediction by incorporating both surface geometric and non-geometric factors. Trained and tested on three prominent antibody-antigen benchmarks, ParaSurf achieves state-of-the-art results across nearly all metrics. Unlike models restricted to the variable region, ParaSurf demonstrates the ability to accurately predict binding scores across the entire Fab region of the antibody. Additionally, we conducted an extensive analysis using the largest of the three datasets employed, focusing on three key components: (i) a detailed evaluation of paratope prediction for each complementarity-determining region loop, (ii) the performance of models trained exclusively on the heavy chain, and (iii) the results of training models solely on the light chain without incorporating data from the heavy chain. AVAILABILITY AND IMPLEMENTATION: Source code for ParaSurf, along with the datasets used, preprocessing pipeline, and trained model weights, are freely available at https://github.com/aggelos-michael-papadopoulos/ParaSurf.

Abstract Adversarial attacks pose a threat to neural networks, requiring robust methods to mitigate them. Adversarial Training has emerged as a promising approach; however, its practical application in real-world deep learning systems is hindered by the trade-offs between efficiency and robustness, as optimizing for one aspect may come at cost of the other. This paper presents a comprehensive investigation into the impact of different Adversarial Training approaches and model types on the robustness of adversarially trained models, while considering the dynamic trade-offs involved. Leveraging our previously published method, Delayed Adversarial Training with Non-Sequential Adversarial Epochs – DATNS, we conduct extended empirical analyses through new experiments to effectively balance these trade-offs and navigate the interplay between efficiency and robustness, as well as catastrophic forgetting and interpretability. By providing our insights on the discussed trade-offs this research aims to enable the development of more efficient, robust, and interpretable models against adversarial attacks.

In this work, a novel federated distillation weight aggregation method is proposed. Specifically, an algorithm designed for effective learning in distributed environments is introduced. This algorithm includes an innovative federated distillation scheme, proposing a sophisticated aggregation of model outputs, employing a global server model to manage process. On the server side, feature mixing is employed to aggregate client representations before they are transmitted back to the client side for knowledge distillation. During feature mixing a weight factor is assigned to each client’s logits, penalizing bad quality clients and preserving system’s credibility. Thorough experimentation has been conducted to comprehensively study the issue at hand. Key findings reveal the significant potential of the proposed solution, which achieves robust performance in a federated setting while reducing communication costs.

In this work, a novel federated distillation weight aggregation method is proposed. Specifically, an algorithm designed for effective learning in distributed environments is introduced. This algorithm includes an innovative federated distillation scheme, proposing a sophisticated aggregation of model outputs, employing a global server model to manage process. On the server side, feature mixing is employed to aggregate client representations before they are transmitted back to the client side for knowledge distillation. During feature mixing a weight factor is assigned to each client’s logits, penalizing bad quality clients and preserving system’s credibility. Thorough experimentation has been conducted to comprehensively study the issue at hand. Key findings reveal the significant potential of the proposed solution, which achieves robust performance in a federated setting while reducing communication costs.

This paper addresses the challenge of reducing annotation costs in distributed learning environments, particularly in systems with limited data and computational resources, such as those found in edge devices. We propose Federated Reinforced Active Learning, a framework that integrates Federated Learning with Reinforced Active Learning to optimize data labeling under strict cost constraints. The method is designed for small-scale networks where data is sparse, and minimal training epochs are available. By utilizing reinforcement learning within active learning, the system selects the most informative data samples, allowing for efficient training while significantly reducing the need for extensive annotations. This approach is particularly suited for environments where minimizing both annotation and computational costs is critical, such as in applications where cost efficiency and resource limitations are top priorities. The proposed method is evaluated on the CIFAR-10 and CIFAR-100 datasets using ResNet18, across 5 and 10 clients. Results demonstrate that the method significantly reduces annotation costs and improves learning outcomes, making it an ideal solution for cost-sensitive distributed systems.

Recent technological advances in the fields of data and computer science have improved significantly the everyday life of people. However, technological advances are also being adopted by criminals to facilitate and expand their illicit actions. The Deep Learning (DL) paradigm has shown a significant potential in analysing complex structured data. However, in the crime detection domain, a limited number of public datasets is available, constrained to specific tasks only, which hinders the research and development of accurate and robust DL-assisted tools. The goal of this work is to extend the well-known UCF-crime dataset to the case of video captioning. To the best of our knowledge, this is the first publicly available crime-related video captioning dataset. A new proposed video captioning approach is compared to a plethora of state-of- the-art-methods in this dataset, while qualitative and quantitative characteristics of the latter are presented.

In this paper, a comparative experimental assessment of computer vision-based methods for sign language recognition is conducted. By implementing the most recent deep neural network methods in this field, a thorough evaluation on multiple publicly available datasets is performed. The aim of the present study is to provide insights on sign language recognition, focusing on mapping non-segmented video streams to glosses. For this task, two new sequence training criteria, known from the fields of speech and scene text recognition, are introduced. Furthermore, a plethora of pretraining schemes is thoroughly discussed. Finally, a new RGB+D dataset for the Greek sign language is created. To the best of our knowledge, this is the first sign language dataset where three annotation levels are provided (individual gloss, sentence and spoken language) for the same set of video captures.

This paper presents the NAct (Nutrition & Activity) Ontology, designed to drive personalised nutritional and physical activity recommendations and effectively support healthy living, through a reasoning-based AI decision support system. NAct coalesces nutritional, medical, behavioural and lifestyle indicators with potential dietary and physical activity directives. The paper presents the first version of the ontology, including its co-design and engineering methodology, along with usage examples in supporting healthy nutritional and physical activity choices. Lastly, the plan for future improvements and extensions is discussed.

In the field of 3D Human Performance Capture, a high-quality 3D scan of the performer is rigged and skinned to an animatable 3D template mesh that is subsequently fitted to the captured performance's RGB-D data. Template fitting is accomplished via solving for the template's pose parameters that better explain the performance data at each recorded frame. In this paper, we challenge open implementations of zeroth-order optimizers to solve the template fitting problem in a human performance capture dataset. The objective function that we employ approximates, the otherwise costly to evaluate, 3D RMS hausdorff distance between the animated template and the 3D mesh reconstructed from the depth data (target mesh) at an individual recorded frame. We distinguish and benchmark the optimizers, in three different real-world scenarios, two of which are based on the geometric proximity of the template to the target in individual frames, while in the third one we fit the template sequentially to all target frames of the recorded sequence. Conclusions of this work can serve as a reference for future optimizer implementations and our findings can server as a baseline for future multi-objective optimization approaches. We make part of our benchmark and experiment setup publicly available (https://github.com/VCL3D/nevergrad, https://github.com/VCL3D/PerformanceCapture/releases/).

Adversarial Training (AT) has so far been considered as the most effective approach for building robust computer vision architectures that can adequately handle adversarial attacks. However, its high complexity, rising from the need to repeatedly generate attacks against image batches over training epochs, has been a controlling factor to its generalized use. Therefore, to date, AT has only been used for training on vanilla datasets, casting doubt on how soon it can be applied to real world computer vision systems in safety critical domains. Few researchers have to a certain extent addressed the issue by proposing AT variations which have moderately improved the complexity. Extending these advances, this work explores simple yet effective ways to further improve the complexity of AT in terms of training time. Specifically, it is explored through this work whether a) training sequential adversarial epochs and b) attacking the entire dataset during adversarial epochs are necessary for a robust learned model.

Communication with a customer or future user during the planning and design phase is crucial in applications such as interior design and furniture retailing. Augmented Reality (AR) has the potential to make these communication processes highly effective and provide a better experience for the customer. Current AR authoring solutions are quite complex and require manually creating scenes or rely on objects prepared with even more complex applications such as CAD tools. However, both design experts and their customers often lack the IT skills to use these tools. In addition, many practical cases involve changing reality rather than just adding to it, thus requiring the use of Diminished Reality (DR) technologies. This paper presents a comprehensive analysis of the requirements of both professionals and consumers (gathered using user surveys and individual interviews) for a lightweight and automated authoring process of AR and DR experiences, deriving a set of requirements that can be aligned with state of the art technologies and identifying a number of challenges for AR research.

MOTIVATION: The knowledge of potentially druggable binding sites on proteins is an important preliminary step toward the discovery of novel drugs. The computational prediction of such areas can be boosted by following the recent major advances in the deep learning field and by exploiting the increasing availability of proper data. RESULTS: In this article, a novel computational method for the prediction of potential binding sites is proposed, called DeepSurf. DeepSurf combines a surface-based representation, where a number of 3D voxelized grids are placed on the protein's surface, with state-of-the-art deep learning architectures. After being trained on the large database of scPDB, DeepSurf demonstrates superior results on three diverse testing datasets, by surpassing all its main deep learning-based competitors, while attaining competitive performance to a set of traditional non-data-driven approaches. AVAILABILITY AND IMPLEMENTATION: The source code of the method along with trained models are freely available at https://github.com/stemylonas/DeepSurf.git. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

It has been shown that global scene understanding tasks like layout estimation can benefit from wider field of views, and specifically spherical panoramas. While much progress has been made recently, all previous approaches rely on intermediate representations and postprocessing to produce Manhattan-aligned estimates. In this work we show how to estimate full room layouts in a single-shot, eliminating the need for postprocessing. Our work is the first to directly infer Manhattan-aligned outputs. To achieve this, our data-driven model exploits direct coordinate regression and is supervised end-to-end. As a result, we can explicitly add quasi-Manhattan constraints, which set the necessary conditions for a homography-based Manhattan alignment module. Finally, we introduce the geodesic heatmaps and loss and a boundary-aware center of mass calculation that facilitate higher quality keypoint estimation in the spherical domain. Our models and code are publicly available at https://github.com/VCL3D/SingleShotCuboids.

Drug discovery involves extremely costly and time consuming procedures and can be significantly benefited by computational approaches, such as virtual screening (VS). Structure-based VS relies on scoring functions which aim to evaluate the binding of a candidate compound (ligand) on a protein target. Over the last few years, the advancement of the deep learning field has led to the development of novel scoring functions based on convolutional neural networks (CNN), which have achieved state-of-the-art results. In this paper, we present an integrated end-to-end VS pipeline for application on real-world drug discovery scenarios. It combines multiple conformations of the ligand with a new CNN scoring function based on the ResNet architecture, called ResNetVS, which incorporates also the docking output score in its evaluation. After experiments on the DUD-E dataset, it has shown notable performance, especially in early enrichment, where it overcomes current benchmarks. The proposed pipeline is finally applied on the emerging case of COVID-19 pandemic, in a struggle to discover inhibitors for the viral spike protein-ACE2 interaction.

Learning to understand and infer object functionalities is an important step towards robust visual intelligence. Significant research efforts have recently focused on segmenting the object parts that enable specific types of human-object interaction, the so-called "object affordances". However, most works treat it as a static semantic segmentation problem, focusing solely on object appearance and relying on strong supervision and object detection. In this paper, we propose a novel approach that exploits the spatio-temporal nature of human-object interaction for affordance segmentation. In particular, we design an autoencoder that is trained using ground-truth labels of only the last frame of the sequence, and is able to infer pixel-wise affordance labels in both videos and static images. Our model surpasses the need for object labels and bounding boxes by using a soft-attention mechanism that enables the implicit localization of the interaction hotspot. For evaluation purposes, we introduce the SOR3D-AFF corpus, which consists of human-object interaction sequences and supports 9 types of affordances in terms of pixel-wise annotation, covering typical manipulations of tool-like objects. We show that our model achieves competitive results compared to strongly supervised methods on SOR3D-AFF, while being able to predict affordances for similar unseen objects in two affordance image-only datasets.

Food analysis resides at the core of modern nutrition recommender systems, providing the foundation for a high-level understanding of users' eating habits. This paper focuses on the sub-task of ingredient recognition from food images using a variational framework. The framework consists of two variational encoder-decoder branches, aimed at processing information from different modalities (images and text), as well as a variational mapper branch, which accomplishes the task of aligning the distributions of the individual branches. Experimental results on the Yummly-28K data-set showcase that the proposed framework performs better than similar variational frameworks, while it surpasses current state-of-the-art approaches on the large-scale Recipe1M data-set.

Convolutional Neural Networks (CNNs) have recently been introduced for addressing copy-move forgery detection (CMFD). However, current CMFD CNN-based approaches have insufficient performance commitment regarding the localization of the positive class. In this paper, this issue is explored by considering both linear and nonlinear interactions between pixels. A nonlinear Inception module based on second-order Volterra kernels is proposed, in order to ameliorate the results of a state-of-the-art CMFD architecture. The outcome of this work shows that a combination of linear and nonlinear convolution kernels can make the input foreground and background pixels more separable. The proposed approach is evaluated on CASIA and CoMoFoD, two publicly available CMFD datasets, and results to an improved positive class localization performance. Moreover, the findings of the proposed method imply that the nonlinear Inception module stimulates immense robustness against miscellaneous post processing attacks.

With the advent of consumer grade depth sensors, low-cost volumetric capture systems are easier to deploy. Their wider adoption though depends on their usability and by extension on the practicality of spatially aligning multiple sensors. Most existing alignment approaches employ visual patterns, e.g. checkerboards, or markers and require high user involvement and technical knowledge. More user-friendly and easier-to-use approaches rely on markerless methods that exploit geometric patterns of a physical structure. However, current SoA approaches are bounded by restrictions in the placement and the number of sensors. In this work, we improve markerless data-driven correspondence estimation to achieve more robust and flexible multi-sensor spatial alignment. In particular, we incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one. This is accomplished by a soft, differentiable procrustes analysis that regularizes the segmentation and achieves higher extrinsic calibration performance in expanded sensor placement configurations, while being unrestricted by the number of sensors of the volumetric capture system. Our model is experimentally shown to achieve similar results with marker-based methods and outperform the mark-erless ones, while also being robust to the pose variations of the calibration structure. Code and pretrained models are available at https://vcl3d.github.io/StructureNet/.

Omnidirectional vision is becoming increasingly relevant as more efficient 360° image acquisition is now possible. However, the lack of annotated 360° datasets has hindered the application of deep learning techniques on spherical content. This is further exaggerated on tasks where ground truth acquisition is difficult, such as monocular surface estimation. While recent research approaches on the 2D domain overcome this challenge by relying on generating normals from depth cues using RGB-D sensors, this is very difficult to apply on the spherical domain. In this work, we address the unavailability of sufficient 360° ground truth normal data, by leveraging existing 3D datasets and remodelling them via rendering. We present a dataset of 360° images of indoor spaces with their corresponding ground truth surface normal, and train a deep convolutional neural network (CNN) on the task of monocular 360° surface estimation. We achieve this by minimizing a novel angular loss function defined on the hyper-sphere using simple quaternion algebra. We put an effort to appropriately compare with other state of the art methods trained on planar datasets and finally, present the practical applicability of our trained model on a spherical image re-lighting task using completely unseen data by qualitatively showing the promising generalization ability of our dataset and model.

Learning based approaches for depth perception are limited by the availability of clean training data. This has led to the utilization of view synthesis as an indirect objective for learning depth estimation using efficient data acquisition procedures. Nonetheless, most research focuses on pinhole based monocular vision, with scarce works presenting results for omnidirectional input. In this work, we explore spherical view synthesis for learning monocular 360 depth in a self-supervised manner and demonstrate its feasibility. Under a purely geometrically derived formulation we present results for horizontal and vertical baselines, as well as for the trinocular case. Further, we show how to better exploit the expressiveness of traditional CNNs when applied to the equirectangular domain in an efficient manner. Finally, given the availability of ground truth depth data, our work is uniquely positioned to compare view synthesis against direct supervision in a consistent and fair manner. The results indicate that alternative research directions might be better suited to enable higher quality depth perception. Our data, models and code are publicly available at https://vcl3d.github.io/SphericalViewSynthesis/.

This paper presents the second edition of the “drone-vs-bird” detection challenge, launched within the activities of the 16-th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS). The challenge's goal is to detect one or more drones appearing at some point in video sequences where birds may be also present, together with motion in background or foreground. Submitted algorithms should raise an alarm and provide a position estimate only when a drone is present, while not issuing alarms on birds, nor being confused by the rest of the scene. This paper reports on the challenge results on the 2019 dataset, which extends the first edition dataset provided by the SafeShore project with additional footage under different conditions.

The popularity of Unmanned Aerial Vehicles (UAVs) is increasing year by year and reportedly their applications hold great shares in global technology market. Yet, since UAVs can be also used for illegal actions, this raises various security issues that needs to be encountered. Towards this end, UAV detection systems have emerged to detect and further anticipate inimical drones. A very significant factor is the maximum detection range in which the system's senses can “see” an upcoming UAV. For those systems that employ optical cameras for detecting UAVs, the main issue is the accurate drone detection when it fades away into sky. This work proposes the incorporation of Super-Resolution (SR) techniques in the detection pipeline, to increase its recall capabilities. A deep SR model is utilized prior to the UAV detector to enlarge the image by a factor of 2. Both models are trained in an end-to-end manner to fully exploit the joint optimization effects. Extensive experiments demonstrate the validity of the proposed method, where potential gains in the detector's recall performance can reach up to 32.4%.

It is a challenge to aggregate and analyze data from heterogeneous social media sources not only for businesses and organizations but also for Law Enforcement Agencies. The latter’s core objectives are to monitor criminal and terrorist related activities and to identify the ”key players” in various networks. In this paper, a framework for homogenizing and exploiting data from multiple sources is presented. Moreover, as part of the framework, an ontology that reflects today’s social media perceptions is introduced. Data from multiple sources is transformed into a labeled property graph and stored in a graph database in a homogenized way based on the proposed ontology. The result is a cross-source analysis system where end-users can explore different scenarios and draw conclusions through a library of predefined query placeholders that focus on forensic investigation. The framework is evaluated on the Stormfront dataset, a radical right, web community. Finally, the benefits of applying the proposed framework to discover and visualize the relationships between the Stormfront profiles are presented.

Depth perception is considered an invaluable source of information for various vision tasks. However, depth maps acquired using consumer-level sensors still suffer from non-negligible noise. This fact has recently motivated researchers to exploit traditional filters, as well as the deep learning paradigm, in order to suppress the aforementioned non-uniform noise, while preserving geometric details. Despite the effort, deep depth denoising is still an open challenge mainly due to the lack of clean data that could be used as ground truth. In this paper, we propose a fully convolutional deep autoencoder that learns to denoise depth maps, surpassing the lack of ground truth data. Specifically, the proposed autoencoder exploits multiple views of the same scene from different points of view in order to learn to suppress noise in a self-supervised end-to-end manner using depth and color information during training, yet only depth during inference. To enforce self-supervision, we leverage a differentiable rendering technique to exploit photometric supervision, which is further regularized using geometric and surface priors. As the proposed approach relies on raw data acquisition, a large RGB-D corpus is collected using Intel RealSense sensors. Complementary to a quantitative evaluation, we demonstrate the effectiveness of the proposed self-supervised denoising approach on established 3D reconstruction applications. Code is avalable at https://github.com/VCL3D/DeepDepthDenoising.

Residual networks (ResNets) have introduced a milestone for the deep learning community due to their outstanding performance in diverse applications. They enable efficient training of increasingly deep networks, reducing the training difficulty and error. The main intuition behind them is that, instead of mapping the input information, they are mapping a residual part of it. Since the original work, a lot of extensions have been proposed to improve information mapping. In this paper, a novel extension of the residual block is proposed inspired by linear dynamical systems (LDSs), called LDS-ResNet. Specifically, a new module is presented that improves mapping of residual information by transforming it in a hidden state and then mapping it back to the desired feature space using convolutional layers. The proposed module is utilized to construct multi-branch residual blocks for convolutional neural networks. An exploration of possible architectural choices is presented and evaluated. Experimental results show that LDS-ResNet outperforms the original ResNet in image classification and object detection tasks on public datasets such as CIFAR-10/100, ImageNet, VOC, and MOT2017. Moreover, its performance boost is complementary to other extensions of the original network such as pre-activation and bottleneck, as well as stochastic training and Squeeze-Excitation.

The present work investigates the use of 3D flow information for performing Deep Learning (DL)-based human action recognition. Generally, 3D flow fields include rich and fine-grained information, regarding the motion dynamics of the observed human actions. However, despite the great potentials present, 3D flow has not been widely used, mainly due to challenges related to the efficient modeling of the flow information and the addressing of the respective computational complexity issues. In this paper, different techniques are investigated for incorporating 3D flow information in DL action recognition schemes. In particular, a novel sequence modeling approach is introduced, which combines the advantageous characteristics for spatial correlation estimation of Convolutional Neural Networks (CNNs) with the increased temporal modeling capabilities of Long Short Term Memory (LSTM) models. Additionally, an extended CNN - based deep flow model is proposed that extracts features from both the spatial and temporal domains, by applying 3D convolutions; hence, modeling the action dynamics within consecutive frames. Moreover, for compact and efficient 3D motion feature extraction, the combined use of CNNs with a ‘flow colorization’ approach is adopted. The proposed methods significantly outperform similar DL and hand-crafted 3D flow approaches, and compare favorably with most skeleton-based techniques in the currently most challenging public dataset, namely the NTU RGB-D.

Sensorimotor learning, namely the process of understanding the physical world by combining visual and motor information, has been recently investigated, achieving promising results for the task of 2D/3D object recognition. Following the recent trend in computer vision, powerful deep neural networks (NNs) have been used to model the “sensory” and “motor” information, namely the object appearance and affordance. However, the existing implementations cannot efficiently address the spatio-temporal nature of the human-object interaction. Inspired by recent work on attention-based learning, this paper introduces an attention-enhanced NN-based model that learns to selectively focus on parts of the physical interaction where the object appearance is corrupted by occlusions and deformations. The model's attention mechanism relies on the confidence of classifying an object based solely on its appearance. Three metrics are used to measure the latter, namely the prediction entropy, the average N-best likelihood difference, and the N-best likelihood dispersion. Evaluation of the attention-enhanced model on the SOR3D dataset reports 33% and 26% relative improvement over the appearance-only and the spatio-temporal fusion baseline models, respectively.

In this work, the structure of the moving skeleton, which is a time varying graph, along with the temporal dependencies of human action were leveraged in the scope of skeleton action recognition. The optimization of the proposed model shares similarities with the optimization problem of Slow Feature Analysis (SFA) enabling a well defined solution. Moreover, due to the incorporated skeleton structure, the learned slow functions enclose information regarding the geometry of the skeleton movement which is very useful in the action recognition problem. Two skeleton action datasets were used to evaluate our method, the MSR Action 3D and a dataset whose actions were inspired by psychological studies. Both datasets were captured by depth cameras. The proposed method yielded promising results when evaluated on the aforementioned datasets.

Geometry and texture resolution are two common system parameters of any modern volumetric 3D reconstruction pipeline. In tele-immersive applications, besides their apparent impact on the visual quality of the output 3D mesh, their absolute values implicitly influence the computational load of the whole tele-immersion pipeline from acquisition to 3D reconstruction, compression and transmission. Thus, tuning those parameters to an optimal combination has evident benefits. In this paper, we conduct a subjective experiment to assess the visual quality of textured human 3D-reconstructed meshes that are produced by a volumetric 3D reconstruction algorithm as a joint function of the geometry and texture resolution production parameters. The experiment is based on the forced choice pairwise comparison methodology on pre-rendered views of the real-time reconstructed meshes within the context of human performance capture. We analyze the pairwise comparison data and establish a ranking of the parameter space and, thus also, a mapping from the parameters to the subjective visual quality. The results of this study may be utilized to tune the parameters of the real-time 3D reconstruction pipeline, optimizing for the best balance between visual quality, bandwidth and overall performance.

Traditional VR is mostly about headset experiences either in completely virtual environments or 360° videos. On the other hand AR has been mixing realities by inserting the virtual within the real. In this work we present the Augmented VR concept that lies at the middle right of the virtuality continuum, typically referred to as augmented virtuality. We offer another perspective by blending the real within the virtual focusing on capturing actual human performances in three dimensions and emplacing them within virtual environments [1]-[3]. By compressing and transmitting this new type of 3D media we can also achieve real-time interaction, communication and collaboration between users. Being in full 3D our media are compatible with a variety of applications be it either VR, AR, MR and open up new exciting opportunities like free viewpoint spectating while also increasing the feeling of immersion of all participating users. We demonstrate our technology via a prototype two player game that can support spectating in various devices like head mounted displays (VR) or tablet laptops (AR). Our system is easy to setup, requiring minimal non-technical human intervention, and relatively low cost taking one step ahead in making this technology available to the consumer public.

In this paper, we address the problem of recognizing student engagement in prosocial games by exploiting engagement cues from different input modalities. Since engagement is a multifaceted phenomenon with different dimensions, i.e., behavioral, cognitive, and affective, we propose the modeling of student engagement using real-time data from both the students and the game. More specifically, we apply body motion and facial expression analysis to identify the affective state of students, while we extract features related to their cognitive and behavioral engagement based on the analysis of their interaction with the game. For the automatic recognition of engagement, we adopt a machine learning approach based on artificial neural networks, while for the annotation of the engagement data, we introduce a novel approach based on the use of games with different degrees of challenge in conjunction with a retrospective self-reporting method. To evaluate the proposed methodology, we conducted real-life experiments in 4 classes, in 3 primary schools, with 72 students and 144 gameplay recordings in total. Experimental results show the great potential of the proposed methodology, which improves the classification accuracy of the three distinct dimensions with a detection rate of 85%. A detailed analysis of the role of each component of the Game Engagement Questionnaire, i.e., immersion, presence, flow, and absorption, in the classification process is also presented in this paper.

In this paper, the problem of human action recognition using 3D reconstruction data is deeply investigated. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of this paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, this paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect).

The analysis of multimodal data collected by innovative imaging sensors, Internet of Things (IoT) devices and user interactions, can provide smart and automatic distant monitoring of patients and reveal valuable insights for early detection and/or prevention of events related to their health situation. In this paper, we present a platform called ICT4LIFE which starting from low-level data capturing and performing multimodal fusion to extract relevant features, can perform high-level reasoning to provide relevant data on monitoring and evolution of the patient, and trigger proper actions for improving the quality of life of the patient.

In this paper, a novel multi-modal method for person identification in indoor environments is presented. This approach relies on matching the skeletons detected by a Kinect v2 device with wearable devices equipped with inertial sensors. Movement features such as yaw and pitch changes are employed to associate a particular Kinect skeleton to a person using the wearable. The entire process of sensor calibration, feature extraction, synchronization and matching is detailed in this work. Six detection scenarios were defined to assess the proposed method. Experimental results have shown a high accuracy in the association process.

Moving objects pose a challenge to every video stabilization algorithm. We present a novel, efficient filtering technique that manages to remove outlier motion vectors caused from moving objects in a per-pixel smoothing setting. We leverage semantic information to change the calculation of optical flow, forcing the outliers to reside in the edges of our semantic mask. After a `content-preserving warping' and a smoothing step we manage to produce stable and artifact-free videos.

The ICT4Life Open Source framework contains libraries for acquiring and processing data from different sensors, machine learning algorithms for activity recognition, as well as fusion methods of multiple modalities either at an early or at a late stage. The main purpose of the introduced system is to enable an easy customization of patients' monitoring using different types of sensors. Furthermore, by allowing an easy integration of new sensors or types of activities, the proposed subsystem supports the development of new solutions for different diseases, than the ones considered in the ICT4Life p roject.

Cardiac Rehabilitation (CR) can significantly improve mortality and morbidity rates from Cardiovascular Diseases (CVD). Nevertheless, traditional CR is diminished by low subsequent adherence rates. Thus, in this paper, an e-Health technological module for human motion analysis and user modelling is proposed, in order to address the requirements of unsupervised, tele-rehabilitation systems for CVD, by evaluating and personalizing prescribed physical CR programs. The proposed module consists of a) an exercise capturing and evaluation component, and b) a user modelling and decision support system for personalization of cardiac rehabilitation programs. In particular, the module monitors and analyses the body movements of the patient when exercising in real-time, while based on this analysis and the heart-rate measurements, it is capable of short-term and long-term CR session adaptation. The proposed module constitutes a significant tool for internet-enabled sensor-based home exercise platforms.

It is well-established by cognitive neuroscience that human perception of objects constitutes a complex process, where object appearance information is combined with evidence about the so-called object affordances, namely the types of actions that humans typically perform when interacting with them. This fact has recently motivated the sensorimotor approach to the challenging task of automatic object recognition, where both information sources are fused to improve robustness. In this work, the aforementioned paradigm is adopted, surpassing current limitations of sensorimotor object recognition research. Specifically, the deep learning paradigm is introduced to the problem for the first time, developing a number of novel neuro-biologically and neuro-physiologically inspired architectures that utilize state-of-the-art neural networks for fusing the available information sources in multiple ways. The proposed methods are evaluated using a large RGB-D corpus, which is specifically collected for the task of sensorimotor object recognition and is made publicly available. Experimental results demonstrate the utility of affordance information to object recognition, achieving an up to 29% relative error reduction by its inclusion.

During the last years, Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in image classification. Their architectures have largely drawn inspiration by models of the primate visual system. However, while recent research results of neuroscience prove the existence of non-linear operations in the response of complex visual cells, little effort has been devoted to extend the convolution technique to non-linear forms. Typical convolutional layers are linear systems, hence their expressiveness is limited. To overcome this, various non-linearities have been used as activation functions inside CNNs, while also many pooling strategies have been applied. We address the issue of developing a convolution method in the context of a computational model of the visual cortex, exploring quadratic forms through the Volterra kernels. Such forms, constituting a more rich function space, are used as approximations of the response profile of visual cells. Our proposed second-order convolution is tested on CIFAR-10 and CIFAR-100. We show that a network which combines linear and non-linear filters in its convolutional layers, can outperform networks that use standard linear filters with the same architecture, yielding results competitive with the state-of-the-art on these datasets.

Camera pose estimation is a fundamental problem of Augmented Reality and 3D reconstruction systems. Recently, despite the new better performing direct methods being developed, state-of-the-art methods are still estimating erroneous poses due to sensor noise, environmental conditions and challenging trajectories. Adding a back-end mapping process, SLAM systems achieve better performance and are more robust, but require higher computational resources, limiting their applicability. Therefore, lighter solutions to improve the accuracy of pose estimates are required. In this work we demonstrate the effectiveness of lighter data structures, namely surface elements, and exploit the temporality of sensor data streams to accumulate moving camera frames and improve tracking. This representation allows us to splat a photometric and geometric model simultaneously and use it to improve the performance of dense RGB-D camera pose estimation methods. Exploiting Elliptical Weighted Average splatting to produce high quality photometric results also allows us to detect erroneous poses through a novel visual quality analysis process. We show evidence of the EWA temporal model's effectiveness in publicly available datasets and argue that point-based representations are a good candidate for building lighter systems that should be further explored.

Human activity recognition has received a lot of attention recently, mainly thanks to the advancements in sensing technologies and systems’ increasing computational power. However, complexity in human movements, sensing devices’ noise and person-specific characteristics impose challenges that still remain to be overcome. In the proposed work, a novel, multi-modal human action recognition method is presented for handling the aforementioned issues. Each action is represented by a basis vector and spectral analysis is performed on an affinity matrix of new action feature vectors. Using modality-dependent kernel regressors for computing the affinity matrix, complexity is reduced and robust low-dimensional representations are achieved. The proposed scheme supports online adaptivity of modalities, in a dynamic fashion, according to their automatically inferred reliability. Evaluation on three publicly available datasets demonstrates the potential of the approach.

The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton-based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors' calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture-mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies.

Real-world CCTV footage often poses increased challenges in object tracking due to Pan-Tilt-Zoom operations, low camera quality and diverse working environments. Most relevant challenges are moving background, motion blur and severe scale changes. Convolutional neural networks, which offer state-of-the-art performance in object detection, are increasingly utilized to pursue a more efficient tracking scheme. In this work, the use of heterogeneous training data and data augmentation is explored to improve their detection rate in challenging CCTV scenes. Moreover, it is proposed to use the objects' spatial transformation parameters to automatically model and predict the evolution of intrinsic camera parameters and accordingly tune the detector for better performance. The proposed approaches are tested on publicly available datasets and real-world CCTV videos.

Convolutional Neural Networks (CNNs), which have nowadays dominated image analysis tasks, constitute feed-forward methods that model increasingly complex data structures and patterns along the subsequent hidden layers of the network. However, the common practice of using the activation features from the last network layer inevitably leads to a visual recognition bottleneck. This is due to the fact that discriminative features for different objects of varying complexity do not need to be extracted from the same layer. To this end, a novel frequency domain analysis of the feature maps of the same as well as of different network layers is proposed. In this way, the proposed method exploits more efficiently the knowledge that is stored in the actual CNN and facilitates in identifying the most discriminative features for every individual object type. Experimental results in a large-scale real-world Closed-Circuit Television (CCTV) surveillance and the PASCAL VOC 2012 datasets demonstrate the efficiency of the proposed approach.

In this paper, we propose a novel, full-body, real-time 3D reconstruction framework that makes use of pre-scanned body parts (more precisely pre-scanned 3D heads) so as to provide a more detailed 3D reconstruction mainly in the semantically important head area. Our framework deals with 3 major challenges: (a) multiple depth sensors collaboration, (b) pre-scanned head positioning and (c) reconstruction and texturing. In all the above challenges, we propose novel solutions so as to cope with time and space complexity, synchronization and 3D mesh quality. Experimental evaluation provides evidence that superior 3D mesh quality can be achieved compared to simple (not enhanced) use of depth cameras' data.

Tele-immersion (TI) related technologies can change the way people interact and bridge the gap between the physical and digital worlds. However, while the technology itself advances, most developed platforms have complex setups and require large investments. In this work, a low-cost platform is introduced, integrating multiple TI-related advances. Focusing on ease of use and rapid deployment, a fast and fully automatic calibration method is proposed. The platform enables real-time 3D reconstruction of users and their placement into a pre-authored 3D environment. Moreover, interaction is achieved through the user's body posture, removing the need for additional equipment and enabling natural control while immersed. Developing a real-time TI platform requires the efficient integration of several multidisciplinary elements. An elegant, minimal solution to these challenges is proposed and validated in a prototype TI multiplayer game, SpaceWars.

In this paper, a framework for shape-based similarity search of 3D molecular structures is presented. The proposed framework exploits simultaneously the discriminative capabilities of a global, a local, and a hybrid local-global shape feature to produce a geometric descriptor that achieves higher retrieval accuracy than each feature does separately. Global and hybrid features are extracted using pairwise computations of diffusion distances between the points of the molecular surface, while the local feature is based on accumulating pairwise relations among oriented surface points into local histograms. The local features are integrated into a global descriptor vector using the bag-of-features approach. Due to the intrinsic property of its constituting shape features to be invariant to articulations of the 3D objects, the framework is appropriate for similarity search of flexible 3D molecules, while at the same time it is also accurate in retrieving rigid 3D molecules. The proposed framework is evaluated in flexible and rigid shape matching of 3D protein structures as well as in shape-based virtual screening of large ligand databases with quite promising results.

Driven by the needs of customers and industry, online fashion search and analytics are recently gaining much attention. As fashion is mostly expressed by visual content, the analysis of fashion images in online social networks is a rich source of possible insights on evolving trends and customer preferences. Although a plethora of visual content is available, the modeling of clothes’ physics and movement, the implicit semantics in fashion designs, and the subjectivity of their interpretation pose difficulties to fully automated solutions for fashion search and analysis. In this article, we present the design and evaluation of a crowd-powered system for fashion similarity search from Twitter, supporting trend analysis for fashion professionals. The system enables fashion similarity search based on specific human-based similarity criteria. This is achieved by implementing a novel machine--crowd workflow that supports complex tasks requiring highly subjective judgments where multiple true solutions may coexist. We discuss how this leads to a novel class of crowd-powered systems for which the output of the crowd is not used to verify the automatic analysis but is the desired outcome. Finally, we show how this kind of crowd involvement enables a novel kind of similarity search and represents a crucial factor for the acceptance of system results by the end user.

This paper presents a novel approach, which relies on content-based guided image filtering and weighted semi-global optimization for fast and accurate disparity estimation. The approach uses a pixel-based cost term that combines gradient, Gabor-Feature, and color information. The pixel-based matching costs are filtered by applying guided image filtering, which relies on rectangular support windows of two different sizes. In this way, two filtered costs are estimated for each pixel. Among the two filtered costs, the one that will be finally assigned to each pixel depends on the local image content around this pixel. The filtered cost volume is further refined by exploiting weighted semi-global optimization, which improves the disparity estimation accuracy. Finally, the disparity refinement in outlier regions relies on a straightforward and time-efficient outliers handling scheme and on a simple approach which deals with the disparity outliers at depth discontinuities. Experimental results on the Middlebury online stereo evaluation benchmark and 27 additional Middlebury stereo pairs prove that our method is able to generate disparity maps with high accuracy while keeping the computational cost low.

We present a demonstration of a multi-modal 3D capturing platform coupled to a motion comparison system. This work is focused on the preservation of Traditional Sports and Games, namely the Gaelic sports from Ireland and Basque sports from France and Spain. Users can learn, compare and compete in the performance of sporting gestures and compare themselves to real athletes. Our online gesture database provides a way to preserve and display a wide range of sporting gestures. The capturing devices utilised are Kinect 2 sensors and wearable inertial sensors, where the number required varies based on the requested scenario. The fusion of these two capture modalities, coupled to our inverse kinematic algorithm, allow us to synthesize a fluid and reliable 3D model of the user gestures over time. Our novel comparison algorithms provide the user with a performance score and a set of comparison curves (i.e. joint angles and angular velocities), providing a precise and valuable feedback for coaches and players.

We present a novel, low-cost, interactive, exercise-based rehabilitation system. Our research involves the investigation and development of patient-centric, sensor-based rehabilitation games and surrounding technologies. HeartHealth is designed to provide a safe, personalised and fun exercise environment that could be deployed in any exercise based rehabilitation program. HeartHealth utilises a cloud-based patient information management system built on FIWARE Generic Enablers,and motion tracking coupled with our sophisticated motion comparison algorithms. Users can record customised exercises through a doctors interface and then play the rehabilitation game where they must perform a sequence of their exercises in order to complete the game scenario. Their exercises are monitored, recorded and compared by our Motion Evaluation software and real-time feedback is than given based on the users performance.

In this work, we explore the potential of exploiting activity-related global features in order to improve the performance of an existing human Time-Varying Mesh (TVM) compression scheme. The TVM compression scheme used, employs two kinds of frames, namely Intra(I)-Fames and Enhanced Predicted(EP) Frames. In this scheme, I-Frames are used as a reference to encode EP-Frames. The paper introduces a strategy for selecting the most appropriate I-Frame that will serve as a reference frame for the encoding of EP-Frames, exploiting activity-related characteristics. Two different strategies are presented, using a skeleton-matching criterion and a periodicity measurement metric based on human skeleton. Evaluation is conducted on two sequences of the MPEG-3DGC database [1]. Results show that the concept is sound, but they also reveal the sensitivity of the proposed methods to the skeleton quality, thus the need for more robust skeleton tracking techniques.

In this paper the scanline optimization used for stereo matching, is revisited. In order to improve the performance of this semi-global technique, a new criterion to check depth discontinuity, is introduced. This criterion is defined according to the mean-shift-based image segmentation result. Additionally, this work proposes the employment of a pixel dissimilarity metric for the computation of the cost term, which is then provided to the guided image filter approach to estimate the initial cost volume. The algorithm is tested on the four images of the online Middlebury stereo evaluation benchmark. Moreover, it is tested on 27 additional Middlebury stereo pairs for assessing thoroughly its performance. The extended comparison verifies the efficiency of this work.

In this paper, a view-invariant approach to human action recognition using 3D reconstruction data is proposed. Initially, a set of calibrated Kinect sensors are employed for producing a 3D reconstruction of the performing subjects. Subsequently, a 3D flow field is estimated for every captured frame. For performing action recognition, the `Bag-of-Words' methodology is followed, where Spatio-Temporal Interest Points (STIPs) are detected in the 4D space (xyz-coordinates plus time). A novel local-level 3D flow descriptor is introduced, which among others incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every STIP location. Additionally, typical 3D shape descriptors of the literature are used for producing a more complete representation. Experimental results as well as comparative evaluation using datasets from the Huawei/3DLife 3D human reconstruction and action recognition Grand Challenge demonstrate the efficiency of the proposed approach.

In this paper, a novel method for stereo videos disparity estimation is proposed. Temporal consistency in the disparity and color spaces of the video are combined to enhance the disparity estimation algorithm. Outlier detection along the temporal dimension in the color space is used to find areas where disparity can be temporally enhanced from previous and future frames. Moreover, a forward-backward approach is utilized for consistency checking and error correction in cases of fast motion in the video, which usually introduce disparity artifacts. The method can work in real-time even in the hard case of high definition (HD) videos. Experimental results prove that the proposed approach outperforms state of the art methods in a publicly available dataset.

The primary objective of this paper is to present and analyze key aspects related to next-generation tele-immersion applications, studying the end-to-end chain from 3D capturing of remote users to rendering. The key modules for 3D reconstruction of moving humans and their mesh compression, are presented and discussed. The chain performance is evaluated in terms of frame-rates, delay, and visual quality.

In this paper, the problem of real-time, full 3D reconstruction of foreground moving objects, an important task for Tele-Immersion applications, is addressed. More specifically, the proposed reconstruction method receives input from multiple consumer RGB-Depth cameras. A fast and efficient method to calibrate the sensors in initially described. More importantly, an efficient method to smoothly fuse the captured raw point sets is then presented, followed by a volumetric method to produce watertight and manifold meshes. Given the implementation details, the proposed method can operate at high frame rates. The experimental results, with respect to reconstruction quality and rates, verify the effectiveness of the proposed methodology.

This paper proposes a novel methodology for generating 3D point clouds of good accuracy from stereo pairs. Initially, the methodology defines some conditions for the proper selection of image pairs. Then, the selected stereo images are used to estimate dense correspondences using the Daisy descriptor. An efficient two-phase strategy to remove outliers is then introduced. Finally, the 3D point cloud is refined by combining sub-pixel accuracy correspondences estimation and the moving least squares algorithm. The proposed methodology can be exploited by multi-view stereo algorithms due to its good accuracy and its fast computation.

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for real-time transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the state-of-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario.

In this article, a novel method for personalized item recommendation based on social tagging is presented. The proposed approach comprises a content-based tag propagation method to address the sparsity and “cold start” problems, which often occur in social tagging systems and decrease the quality of recommendations. The proposed method exploits (a) the content of items and (b) users' tag assignments through a relevance feedback mechanism in order to automatically identify the optimal number of content-based and conceptually similar items. The relevance degrees between users, tags, and conceptually similar items are calculated in order to ensure accurate tag propagation and consequently to address the issue of “learning tag relevance.” Moreover, the ternary relation among users, tags, and items is preserved by performing tag propagation in the form of triplets based on users' personal preferences and “cold start” degree. The latent associations among users, tags, and items are revealed based on a tensor factorization model in order to build personalized item recommendations. In our experiments with real-world social data, we show the superiority of the proposed approach over other state-of-the-art methods, since several problems in social tagging systems are successfully tackled. Finally, we present the recommendation methodology in the multimodal engine of I-SEARCH, where users' interaction capabilities are demonstrated.

In this paper, the problem of automatic dance performance evaluation from human Motion Capture (MoCap) data is addressed. A novel framework is presented, using data captured by Kinect-based human skeleton tracking, where the evaluation of user's performance is achieved against a gold-standard performance of a teacher. The framework addresses several technical challenges, including global and local temporal synchronization, spatial alignment and comparison of two “dance motion signals.” Towards the solution of these technical challenges, a set of appropriate quaternionic vector-signal processing methodologies is proposed, where the 4D (spatiotemporal) human motion data are represented as sequences of pure quaternions. Such a quaternionic representation offers several advantages, including the facts that joint angles and rotations are inherently encoded in the phase of quaternions and the three coordinates variables ( X,Y,Z) are treated jointly, with their intra-correlations being taken into account. Based on the theory of quaternions, a number of advantageous algorithms are formulated. Initially, global temporal synchronization of dance MoCap data is achieved by the use of quaternionic cross-correlations, which are invariant to rigid spatial transformations between the users. Secondly, a quaternions-based algorithm is proposed for the fast spatial alignment of dance MoCap data. Thirdly, the MoCap data can be temporally synchronized in a local fashion, using Dynamic Time Warping techniques adapted to the specific problem. Finally, a set of quaternionic correlation-based measures (scores) are proposed for evaluating and ranking the performance of a dancer. These quaternions-based scores are invariant to rigid transformations, as proved and demonstrated. A total score metric, through a weighted combination of three different metrics is proposed, where the weights are optimized using Particle Swarm Optimization (PSO). The presented experimental results using the Huawei/3DLife/EMC2dataset are promising and verify the effectiveness of the proposed methods.

In this paper, a gaze-based Relevance Feedback (RF) approach to region-based image retrieval is presented. Fundamental idea of the proposed method comprises the iterative estimation of the real-world objects (or their constituent parts) that are of interest to the user and the subsequent exploitation of this information for refining the image retrieval results. Primary novelties of this work are: a) the introduction of a new set of gaze features for realizing user's relevance assessment prediction at region-level, and b) the design of a time-efficient and effective object-based RF framework for image retrieval. Regarding the interpretation of the gaze signal, a novel set of features is introduced by formalizing the problem under a mathematical perspective, contrary to the exclusive use of explicitly defined features that are in principle derived from the psychology domain. Apart from the temporal attributes, the proposed features also represent the spatial characteristics of the gaze signal, which have not been extensively studied in the literature so far. On the other hand, the developed object-based RF mechanism aims at overcoming the main limitation of region-based RF approaches, i.e., the frequently inaccurate estimation of the regions of interest in the retrieved images. Moreover, the incorporation of a single-camera image processing-based gaze tracker makes the overall system cost efficient and portable. As it is shown by the experimental evaluation, the proposed method outperforms representative global- and region-based explicit RF approaches, using a challenging general-purpose image dataset.

In this paper, a framework for protein-protein docking is proposed, which exploits both shape and physicochemical complementarity to generate improved docking predictions. Shape complementarity is achieved by matching local surface patches. However, unlike existing approaches, which are based on single-patch or two-patch matching, we developed a new algorithm that compares simultaneously, groups of neighboring patches from the receptor with groups of neighboring patches from the ligand. Taking into account the fact that shape complementarity in protein surfaces is mostly approximate rather than exact, the proposed group-based matching algorithm fits perfectly to the nature of protein surfaces. This is demonstrated by the high performance that our method achieves especially in the case where the unbound structures of the proteins are considered. Additionally, several physicochemical factors, such as desolvation energy, electrostatic complementarity (EC), hydrophobicity (HP), Coulomb potential (CP), and Lennard-Jones potential are integrated using an optimized scoring function, improving geometric ranking in more than 60 percent of the complexes of Docking Benchmark 2.4.

The problem of robust, realistic and especially fast 3-D reconstruction of objects, although extensively studied, is still a challenging research task. Most of the state-of-the-art approaches that target real-time applications, such as immersive reality, address mainly the problem of synthesizing intermediate views for given view-points, rather than generating a single complete 3-D surface. In this paper, we present a multiple-Kinect capturing system and a novel methodology for the creation of accurate, realistic, full 3-D reconstructions of moving foreground objects, e.g., humans, to be exploited in real-time applications. The proposed method generates multiple textured meshes from multiple RGB-Depth streams, applies a coarse-to-fine registration algorithm and finally merges the separate meshes into a single 3-D surface. Although the Kinect sensor has attracted the attention of many researchers and home enthusiasts and has already appeared in many applications over the Internet, none of the already presented works can produce full 3-D models of moving objects from multiple Kinect streams in real-time. We present the capturing setup, the methodology for its calibration and the details of the proposed algorithm for real-time fusion of multiple meshes. The presented experimental results verify the effectiveness of the approach with respect to the 3-D reconstruction quality, as well as the achieved frame rates.

In this paper, a novel approximate indexing scheme for efficient content-based image search and retrieval is presented, called Multi-Sort Indexing (MSIDX). The proposed scheme analyzes high dimensional image descriptor vectors, by employing the value cardinalities of their dimensions. The dimensions' value cardinalities, an inherent characteristic of descriptor vectors, are the number of discrete values in the dimensions. As expected, value cardinalities significantly vary, due to the existence of several extraction methods. Moreover, different quantization and normalization techniques used in the extraction process have a strong impact on the dimensions' value cardinalities. Since dimensions with high value cardinalities have more discriminative power, a multiple sort algorithm is used to reorder the descriptors' dimensions according to their value cardinalities, in order to increase the probability of two similar images to lie within a close constant range. The expected bounds of the constant range are defined in detail, following deterministic and probabilistic analyses. The proposed scheme is fully suitable (a) for real-time indexing of images, and (b) for searching and retrieving relevant images with an efficient query processing algorithm. In our experiments with five real datasets, we show the superiority of the proposed approach against hashing methods, also suitable for approximate similarity search.

In this paper, a novel Tensor Factorization and tag Clustering (TFC) model is presented for item recommendation in social tagging systems. The TFC model consists of three distinctive steps, in each of which important innovative elements are proposed. More specifically, through its first step, the content information is exploited to propagate tags between conceptual similar items based on a relevance feedback mechanism, in order to solve sparsity and “cold start” problems. Through its second step, sparsity is further handled, by generating tag clusters and revealing topics, following an innovative tf · idf weighting scheme. Furthermore, we experimentally prove that a few number of expert tags can improve the performance of quality recommendations, since they contribute to more coherent tag clusters. Through its third step, the latent associations among users, topics, and items are revealed by exploiting the TF technique of high order singular value decomposition. This way the proposed TFC model tackles problems of real-world applications, which produce noise and decrease the quality of recommendations. In our experiments with real-world social data, we show that the proposed TFC model outperforms other state-of-the-art methods, which also exploit the TF technique of HOSVD.

Active Contour Models have been widely used in computer vision for segmentation purposes, while anatomically constrained ACMs have offered a valuable solution on medical image segmentation. Efforts have been devoted on various ways of modeling prior knowledge. This paper focuses on how to efficiently incorporate prior knowledge, into an ACM evolution framework, using the structures' distribution map as a second feature image, and blending the two images through a novel adaptive local weighting scheme. For proof of concept the method is applied on hippocampus segmentation in T1-MR brain images, a very challenging task, due to its multivariate surrounding region and the weak, even missing boundaries.

The ever increasing availability of high speed Internet access has led to a leap in technologies that support real-time realistic interaction between humans in online virtual environments. In the context of this work, we wish to realise the vision of an online dance studio where a dance class is to be provided by an expert dance teacher and to be delivered to online students via the web. In this paper we study some of the technical issues that need to be addressed in this challenging scenario. In particular, we describe an automatic dance analysis tool that would be used to evaluate a student's performance and provide him/her with meaningful feedback to aid improvement.

This work addresses the Huawei/3DLife Grand Challenge, presenting a novel method for the analysis of dance movements. The approach focuses on the decomposition of the dance movements into elementary motions. Placing this problem into a probabilistic framework, we propose to exploit Gaussian processes to accurately model the different components of the decomposition. The preliminary results, presented in this paper, are very promising. In particular, two applications are considered, illustrating the relevance of the proposed approach, namely the correction of tracking errors and the smoothing of some movements of the teacher to help toward the dance learning.

This paper proposes a novel framework for 3-D object retrieval, taking into account most of the factors that may affect the retrieval performance. Initially, a novel 3-D model alignment method is introduced, which achieves accurate rotation estimation through the combination of two intuitive criteria, plane reflection symmetry and rectilinearity. After the pose normalization stage, a low-level descriptor extraction procedure follows, using three different types of descriptors, which have been proven to be effective. Then, a novel combination procedure of the above descriptors takes place, which achieves higher retrieval performance than each descriptor does separately. The paper provides also an in-depth study of the factors that can further improve the 3-D object retrieval accuracy. These include selection of the appropriate dissimilarity metric, feature selection/dimensionality reduction on the initial low-level descriptors, as well as manifold learning for re-ranking of the search results. Experiments performed on two 3-D model benchmark datasets confirm our assumption that future research in 3-D object retrieval should focus more on the efficient combination of low-level descriptors as well as on the selection of the best features and matching metrics, than on the investigation of the optimal 3-D object descriptor.

In this paper, a novel framework for rich-media object retrieval is described. The searchable items are media representations consisting of multiple modalities, such as 2-D images, 3-D objects and audio files, which share a common semantic concept. The proposed method utilizes the low-level descriptors of each separate modality to construct a new low-dimensional feature space, where all media objects can be mapped irrespective of their constituting modalities. While most of the existing state-of-the-art approaches support queries of one single modality at a time, the proposed one allows querying with multiple modalities simultaneously, through efficient multimodal query formulation, and retrieves multimodal results of any available type. Finally, a multimedia indexing scheme is adopted to tackle the problem of large scale media retrieval. The present framework proposes significant advances over existing methods and can be easily extended to involve as many heterogeneous modalities as possible. Experiments performed on two multimodal datasets demonstrate the effectiveness of the proposed method in multimodal search and retrieval.

In this paper, a novel approach for fast protein-protein docking based on geometric complementarity is introduced. Complementarity matching is achieved using a rotation-invariant 3D shape descriptor, the Shape Impact Descriptor (SID). Rotation invariance enables matching of equally-sized surface patches without the need for initial alignment. The candidate poses are computed using an intuitive alignment method, which is much faster than exhaustively searching the translational and rotational space of the ligand. The method yields competitive results when compared to other well-known geometry-based, rigid-docking approaches.

This paper presents a novel descriptor for recognizing objects in highly occluded and cluttered 2.5D scenes produced by range scans. This new compact regional shape descriptor, called "projection images", is designed to be robust against noise, partial occlusion and clutter. Projection images are formed by "projections" of points onto the plane centered at the basis point which is perpendicular to the viewing axis. Multiple experiments were performed on a dataset of 50 range scans, each one comprised of multiple objects, which proved that the proposed method is robust and efficient to a satisfactory degree of occlusion and clutter, while it compared favor- ably against descriptors previously introduced in the literature.

This paper presents two algorithms for estimating depth from integral images, which capture a scene by using multiple lenses, offering anaglyph depictions. The first algorithm involves the 3-D integral imaging grid formed by casting rays inversely through the lenses used to capture the integral image. In this formulation, depth estimation is equivalent to finding correspondences on the ray-crossing points. The second algorithm follows the depth-through disparity approach. In this case, a stereo-like minimization problem is formulated which is handled by the graph cuts method. The novelty of the proposed paper lies in constraining the optimization procedures with the “anchor points”. This results in enhanced estimation accuracy, while eliminating the optimization complexity. Anchor points is a set of reliable reference points, detected by applying a robust local image descriptor to viewpoint images, called self-similarity descriptor. The performance of both algorithms is evaluated on a synthetic integral image database in comparison with another state-of-the-art algorithm.

In this paper, a novel framework for 3D object retrieval is presented. The paper focuses on the investigation of an accurate 3D model alignment method, which is achieved by combining two intuitive criteria, the plane reflection symmetry and rectilinearity. After proper positioning in a coordinate system, a set of 2D images (multi-views) are automatically generated from the 3D object, by taking views from uniformly distributed viewpoints. For each image, a set of flip-invariant shape descriptors is extracted. Taking advantage of both the pose estimation of the 3D objects and the flip-invariance property of the extracted descriptors, a new matching scheme for fast computation of 3D object dissimilarity is introduced. Experiments conducted in SHREC 2009 benchmark show the superiority of the pose estimation method over similar approaches, as well as the efficiency of the new matching scheme.

In this work, we describe a novel system that automatically evaluates dance performances against a gold-standard performance and provides visual feedback to the performer in a 3D virtual environment. The system acquires the motion of a performer via Kinect-based human skeleton tracking, making the approach viable for a large range of users, including home enthusiasts. Unlike traditional gaming scenarios, when the motion of a user must by kept in synch with a pre-recorded avatar that is displayed on screen, the technique described in this paper targets online interactive scenarios where dance choreographies can be set, altered, practiced and refined by users. In this work, we have addressed some areas of this application scenario. In particular, a set of appropriate signal processing and soft computing methodologies is proposed for temporally aligning dance movements from two different users and quantitatively evaluating one performance against another.

This paper presents a novel approach for fast rigid docking of proteins based on geometric complementarity. After extraction of the 3D molecular surface, a set of local surface patches is generated based on the local surface curvature. The shape complementarity between a pair of patches is calculated using an efficient shape descriptor, the Shape Impact Descriptor. The key property of the Shape Impact Descriptor is its rotation invariance, which obviates the need for taking an exhaustive set of rotations for each pair of patches. Thus, complementarity matching between two patches is reduced to a simple histogram matching. Finally, a condensed set of almost complementary pairs of surface patches is supplied as input to the final scoring step, where each pose is evaluated using a 3D distance grid. The experimental results prove that the proposed method demonstrates superior performance over other well-known geometry-based, rigid-docking approaches.

Content searching and downloading are the two dominant actions of the Internet users today, despite the fact that the Internet was not originally architected to serve such actions. Content Centric Networking is the new trend in the research community to build network architectures that route content by name and not by the host's network address so as to efficiently deal with content persistence, availability and authenticity issues. In this work we propose an extension to the Content Centric Network protocol in order to support content search as a native process of the network. By using object descriptors to represent the actual content objects and integrate these descriptors in the network protocol we manage to search and retrieve content from the network not only by name but also by the content itself. In this approach, searching for information is a process distributed to the reachable network. Moreover, content aggregation is handled by the end user and not by content aggregation portals, thematic content search engines or information curators. By doing so, search is not any more an application but part of the network and can pave the way for many novel applications which have never been thought until today.

The scale invariant feature transform, SIFT, is one of the most efficient image matching techniques based on local features. It has been applied to various scientific domains such as machine vision, robot navigation, object recognition, etc. In this work, a SIFT improvement is proposed that makes feature matching more robust in the presence of different types of image noise. Thus, Kendall's rank correlation measure is employed to improve the performance of feature matching. Its exploitation reduces the number of erroneous SIFT feature matches without adding significantly to the execution time. The results of the SIFT improvement are validated through matching examples between similar images.

In this paper, a novel transform-based, blind and robust 3-D mesh watermarking scheme is presented. The 3-D surface of the mesh is firstly divided into a number of discrete continuous regions, each of which is successively sampled and mapped onto oblate spheroids, using a novel surface parameterization scheme. The embedding is performed in the spheroidal harmonic coefficients of the spheroids, using a novel embedding scheme. Changes made to the transform domain are then reversed back to the spatial domain, thus forming the watermarked 3-D mesh. The embedding scheme presented herein resembles, in principal, the ones using the multiplicative embedding rule (inherently providing high imperceptibility). The watermark detection is blind and by far more powerful than the various correlators typically incorporated by multiplicative schemes. Experimental results have shown that the proposed blind watermarking scheme is competitively robust against similarity transformations, connectivity attacks, mesh simplification and refinement, unbalanced resampling, smoothing and noise addition, even when juxtaposed to the informed ones.

In this paper, a novel approach for 3-D Shape description and retrieval based on the theory of ellipsoidal harmonics is presented. Four novel descriptors are introduced: the surface ellipsoidal harmonics descriptor, which concerns 3-D objects that are described as polygonal surfaces; the volumetric ellipsoidal harmonics descriptor, which is applicable to volumetric 3-D objects; the generalized ellipsoidal harmonics descriptor that is applied to any local 3-D object descriptors; and, finally, the combined ellipsoidal-spherical harmonics descriptor, which leads to a compact and powerful descriptor that inherits the advantages of both approaches: the rotation invariance properties of the spherical harmonics and the directional information enclosed in ellipsoidal harmonics. Experimental results performed using well-known 3-D object databases prove the retrieval efficiency of the proposed approach.

This paper presents a novel framework for 3-D object content-based search and retrieval, appropriate for both partial and global matching applications. The framework is based on a graph representation of a 3-D object which is enhanced by local geometric features. The 3-D object is decomposed into meaningful parts and an attributed graph is constructed based on the connectivity of the parts. Every 3-D part is approximated with a suitable superellipsoid and a novel 3-D shape descriptor, called a 3-D distance field descriptor, is computed and associated to the corresponding graph nodes. The matching process used is based on attributed graph matching algorithm appropriate for this application. The proposed method not only provides successful retrieval results in terms of geometric similarity but also is invariant to rotation, translation and scaling of an object as well as to the different poses of articulated objects. Finally, it can be effectively used for partial and global 3-D object retrieval.

A novel method for recognizing 3D objects in an occluded, cluttered and noisy 2.5D scene, is presented. A ray-triangle intersection algorithm is used to compute distances between a circular sector that does not belong to the object and a triangulated surface. Firstly, for each sector's point its distance from the object is calculated and stored in a distance map. Secondly, a 2D histogram that counts the distance map's points whose corresponding distance falls within its distance bins, is formed. Then, the percentages of the bin points that fall within each bin are calculated forming the final descriptor vector. The same procedure is followed for the 2.5D scene. The number of the extracted descriptor vectors is independent to the number of the object's or scene's vertices. Experiments proved that the proposed method is fast, robust to noise, occlusion and clutter.

In this paper, a method suitable for partial matching between 3D objects is presented. The 3D objects are firstly segmented into meaningful parts extending a method which is based on the medial surface of the objects. Then, geometric features are extracted for each part using the spherical trace transform. The extracted features are combined and their covariance matrix is computed as a descriptor of each part. The contribution of the proposed approach is that a meaningful segmentation of 3D objects based on medial surface is achieved and that partial matching is performed on meaningful parts, in a rotation, translation and scaling invariant manner. The experimental results performed, proved that the proposed approach achieves accurate partial matching results in terms of distinct meaningful parts as well as satisfactory overall accuracy.

Computational approaches for protein classification have been proposed over the last years in order to speed up the analysis of the biological mechanics in living organisms. Most of the approaches tend to focus in geometrical comparison of the 3D molecules to reach their goals. In this paper a method suitable for partial (sub)graph matching of 3D proteinic models, in order to achieve fast and accurate classification, is proposed. The 3D objects are firstly segmented to their molecular structure. Then, descriptors are extracted for each segment using spherical harmonics algorithms, and graphs are constructed for the molecules. Next, a sub-graph matching procedure is utilized and the results are refined using biochemical properties to get biological meaningful classification. The experimental results proved that the proposed method achieves accurate classification of the proteinic data.

In this paper, a 3D shape-based approach is presented for the efficient search, retrieval, and classification of protein molecules. The method relies primarily on the geometric 3D structure of the proteins, which is produced from the corresponding PDB files and secondarily on their primary and secondary structure. After proper positioning of the 3D structures, in terms of translation and scaling, the Spherical Trace Transform is applied to them so as to produce geometry-based descriptor vectors, which are completely rotation invariant and perfectly describe their 3D shape. Additionally, characteristic attributes of the primary and secondary structure of the protein molecules are extracted, forming attribute-based descriptor vectors. The descriptor vectors are weighted and an integrated descriptor vector is produced. Three classification methods are tested. A part of the FSSP/DALI database, which provides a structural classification of the proteins, is used as the ground truth in order to evaluate the classification accuracy of the proposed method. The experimental results show that the proposed method achieves more than 99 percent classification accuracy while remaining much simpler and faster than the DALI method.

Measuring the similarity between three-dimensional (3-D) objects is a challenging problem, with applications in computer vision, molecular biology, computer graphics, and many other areas. This paper describes a novel method for 3-D model content-based search based on the 3-D Generalized Radon Transform and a querying by-3-D-model approach. A set of descriptor vectors is extracted using the Radial Integration Transform (RIT) and the Spherical Integration Transform (SIT), which represent significant shape characteristics. After the proper alignment of the models, descriptor vectors are produced which are invariant in terms of translation, scaling and rotation. Experiments were performed using three different databases and comparing the proposed method with those most commonly cited in the literature. Experimental results show that the proposed method is adequately satisfactory in terms of both precision versus recall and time needed for retrieval, and that it can be used for 3-D model search and retrieval in a highly efficient manner.

In this paper, the 2D polar wavelet transform is proposed for content based search and retrieval of 3D objects. After the decomposition of a 3D object's volume into a set of planes, the 2D polar wavelet transform is applied to each of them, generating a set of 2D rotation invariant features. These features comprise the input to the spherical trace transform for the final descriptor extraction, which is used for the shape matching. The 2D wavelet transform demonstrated experimentally a high discriminative power, as compared to other existing methods.

In this paper a 3D shape-based approach is presented for the efficient search, retrieval and classification of protein molecules. The method relies on the geometric 3D appearance of the proteins, which is produced from the corresponding PDB files. After proper positioning and alignment of the 3D structures, in terms of translation and scaling, the 3D structures are decomposed into planes. Then, the polar Fourier transform is applied to the planes creating a new domain of concentric spheres. In this new domain a set of functionals is applied so as to produce descriptor vectors, which are completely invariant to rotation and perfectly describe their 3D shape. Experimental results performed using a portion of the FSSP/DALI database shoed that the proposed method achieves more than 98% classification accuracy with less complexity and much simplicity and it is very fast comparing with the DALI method.

This paper describes a novel method for 3D model content-based search and retrieval based on the 3D radon transform and a querying-by-3D-model approach. Descriptors are extracted using the 3D radon transform and applying a number of functionals on the transform's coefficients. Similarity measures are then created for the extracted descriptors and introduced into a 3D model-matching algorithm. This results in a very fast and accurate matching method. Experimental results are presented evaluating the performance of the proposed method in terms of precision versus recall diagrams.

A novel method for the watermarking of 3D models is proposed, which is robust to geometric distortions such its rotation, translation and uniform scaling. After proper positioning and alignment of the 3D models, a watermark is embedded in the vertices of the model using a robust technique for modifying imperceptibly the location of a subset of the 3D model vertices. The watermark can be used as a link to an identifier for the 3D model and the entire system can be used for data hiding applications. One application is the use of watermark as a link to 3D model descriptors for content-based search and retrieval. Experimental results show the ability of the proposed method to the aforementioned attacks. The proposed method is also robust against vertex reordering attack.

This paper presents a novel approach in 3D content-based search and retrieval. First, a set of functional are applied on a 3D model's volume producing a new domain of concentric spheres. In this new domain a new set of functionals is applied, resulting to a completely rotation invariant descriptor vector, which is used for 3D model matching. Experiments were performed using a database and comparing the proposed method with the MPEG-7 3D shape spectrum descriptor. Experimental results show that the proposed method is superior in terms of precision versus recall and can be used for 3D model search and retrieval in a highly efficient manner.