Vision-Language Models (VLMs), exemplified by CLIP, have emerged as foundational for multimodal intelligence. However, their capacity for logical understanding remains significantly underexplored, resulting in critical **''logical blindspots''** that limit their reliability in practical applications. To systematically diagnose this, we introduce **LogicBench**, a comprehensive benchmark with over 50,000 vision-language pairs across 9 logical categories and 4 diverse scenarios: images, videos, anomaly detection, and medical diagnostics. Our evaluation reveals that existing VLMs, even the state-of-the-art ones, fall at over 40 accuracy points below human performance, particularly in challenging tasks like Causality and Conditionality, highlighting their reliance on surface semantics over critical logical structures. To bridge this gap, we propose **LogicCLIP**, a novel training framework designed to boost VLMs' logical sensitivity through advancements in both data generation and optimization objectives. LogicCLIP utilizes logic-aware data generation and a contrastive learning strategy that combines coarse-grained alignment, a fine-grained multiple-choice objective, and a novel logical structure-aware objective. Extensive experiments demonstrate LogicCLIP's substantial improvements in logical comprehension across all LogicBench domains, significantly outperforming baselines. Moreover, LogicCLIP retains, and often surpasses, competitive performance on general vision-language benchmarks, demonstrating that the enhanced logical understanding does not come at the expense of general alignment. We believe LogicBench and LogicCLIP will be important resources for advancing VLM logical capabilities.

Accurate risk assessment is essential for ensuring the safety and efficiency of decision-making processes in autonomous driving systems. Existing studies have primarily focused on modeling traffic environments to capture spatial and interactive relationships among road users. However, achieving human-like scene comprehension remains a formidable challenge. Inspired by human visual cognition mechanisms, this work introduces a novel scanpath-enhanced risk assessment model designed to capture key elements effectively in driving scenarios. The proposed model integrates simulated driver attention distributions and visual behavior patterns, represented through scanpath imagery, to improve critical feature extraction and risk evaluation. Additionally, a specialized node update strategy is presented, specifically aimed at optimizing the ego-car node within heterogeneous traffic contexts. By incorporating simulated attention mechanisms, interactive reasoning, and human drivers’ prior knowledge, the model achieves a more accurate understanding of, and prediction of, the behavioral intentions and interaction dynamics of traffic participants. Extensive experiments on IESG, Non-IESG, and the 1043-Carla datasets demonstrate the superiority of the proposed model compared to existing methods.

Modeling task-driven attention in driving is a fundamental challenge for both autonomous vehicles and cognitive science. Existing methods primarily predict where drivers look by generating spatial heatmaps, but fail to capture the cognitive motivations behind attention allocation in specific contexts, which limits deeper understanding of attention mechanisms. To bridge this gap, we introduce Explainable Driver Attention Prediction, a novel task paradigm that jointly predicts spatial attention regions (where), parses attended semantics (what), and provides cognitive reasoning for attention allocation (why). To support this, we present W3DA, the first large-scale explainable driver attention dataset. It enriches existing benchmarks with detailed semantic and causal annotations across diverse driving scenarios, including normal conditions, safety-critical situations, and traffic accidents. We further propose LLada, a Large Language model-driven framework for driver attention prediction, which unifies pixel modeling, semantic parsing, and cognitive reasoning within an end-to-end architecture. Extensive experiments demonstrate the effectiveness of LLada, exhibiting robust generalization across datasets and driving conditions. This work serves as a key step toward a deeper understanding of driver attention mechanisms, with significant implications for autonomous driving, intelligent driver training, and human-computer interaction.

Despite significant advancements in road event detection (RED), existing approaches encounter critical limitations. These include reliance on single-modal inputs and joint optimization of detection and classification tasks, often leading to conflicting objectives and suboptimal performance. Moreover, their heavy dependence on large-scale annotated datasets restricts generalization in data-scarce scenarios. To address these challenges, we propose AdaRED, a novel framework that decouples agent detection from event classification, thereby mitigating optimization conflicts and enhancing task-specific performance. To comprehensively understand road events, AdaRED uses diverse input modalities, including fine-grained local features, global contextual information, and spatial layout embeddings. Additionally, we introduce the Cross-modal Scene Adaptation Module (CSAM), which integrates lightweight adapters into the multimodal model. This design enables efficient extraction of spatiotemporal features and the integration of visual priors, thereby improving generalization and robustness in challenging scenarios. Extensive experiments on the ROAD-R dataset validate the effectiveness of AdaRED, achieving state-of-the-art performance and addressing the limitations of existing methods. The code can be found on our project page: https://liulinkai.github.io/AdaRED/.

Understanding and decoding human brain activity from electroencephalography (EEG) signals is a fundamental problem in neuroscience and artificial intelligence, with applications ranging from cognition and emotion recognition to clinical diagnosis and brain–computer interfaces. While recent EEG foundation models have made progress in generalized brain decoding by leveraging unified architectures and large-scale pretraining, they inherit a scale-agnostic dense modeling paradigm from NLP and vision. This design overlooks an intrinsic property of neural activity—cross-scale spatiotemporal structure. Different EEG task patterns span a broad range of temporal and spatial scales, from brief neural activations to slow-varying rhythms, and from localized cortical activations to large-scale distributed interactions. Ignoring this diversity may lead to suboptimal representations and weakened generalization ability. To address these limitations, we propose CSBrain, a Cross-scale Spatiotemporal Brain foundation model for generalized EEG decoding. CSBrain introduces two key components: (i) Cross-scale Spatiotemporal Tokenization (CST), which aggregates multi-scale features within localized temporal windows and anatomical brain regions into compact scale-aware token representations; and (ii) Structured Sparse Attention (SSA), which models cross-window and cross-region dependencies for diverse decoding tasks, further enriching scale diversities while eliminating the spurious dependencies. CST and SSA are alternately stacked to progressively integrate cross-scale spatiotemporal dependencies. Extensive experiments across 11 representative EEG tasks and 16 datasets demonstrate that CSBrain consistently outperforms both task-specific models and strong foundation baselines. These results establish cross-scale modeling as a key inductive bias for generalized EEG decoding and highlight CSBrain as a robust backbone for future brain–AI research.

Accurately predicting driver attention is crucial for enhancing advanced driving assistance systems and autonomous vehicles, attracting increasing research interest. Most existing approaches, rooted in general, task-free saliency detection, adopt data-driven paradigms to correlate bottom-up environmental situations with attention distributions. However, they often overlook the complex top-down task-driven aspects of driver attention that are fundamental for the safe navigation of driving tasks, leading to limitations in handling real-world scenarios. In this paper, we take an initial step to explore and introduce BKnet, a Behavior-aware Knowledge-embedded model that innovatively integrates driving behaviors and empirical knowledge. Specifically, inspired by the human long-term cognitive process, we introduce a novel knowledge memory mechanism. It dynamically associates varied traffic scenarios with consistent driving behaviors, fostering the generation of robust behavior-aware empirical knowledge representations. To this end, BKnet facilitates a nuanced and comprehensive simulation of drivers’ attention mechanisms, driven synergistically by both top-down and bottom-up processes. Additionally, we further contribute to the field by collecting a novel Behavior-Aware Driver Attention (BADA) dataset. To the best of our knowledge, BADA is the first attention dataset explicitly incorporated into real-world driving behavior tasks from multiple drivers. Lastly, comprehensive experiments underscore BKnet’s superiority over existing state-of-the-art approaches and validate the effectiveness and necessity of integrating behavior-aware knowledge into driver attention prediction.

Most existing attention prediction research focuses on salient instances like humans and objects. However, the more complex interaction-oriented attention, arising from the comprehension of interactions between instances by human observers, remains largely unexplored. This is equally crucial for advancing human-machine interaction and human-centered artificial intelligence. To bridge this gap, we first collect a novel gaze fixation dataset named IG, comprising 530,000 fixation points across 740 diverse in-teraction categories, capturing visual attention during human observers' cognitive processes of interactions. Subsequently, we introduce the zero-shot interaction-oriented attention prediction task (ZeroIA), which challenges models to predict visual cues for interactions not encountered during training. Thirdly, we present the Interactive Attention model (IA), designed to emulate human observers' cognitive processes to tackle the ZeroIA problem. Extensive experiments demonstrate that the proposed IA outperforms other state-of-the-art approaches in both ZeroIA and fully supervised settings. Lastly, we endeavor to apply interaction-oriented attention to the interaction recognition task itself. Further experimental results demonstrate the promising potential to enhance the performance and inter-pretability of existing state-of-the-art HOI models by incorporating real human attention data from IG and attention labels generated by IA.

Modeling driver attention allocation by scanpath prediction plays a crucial role in advancing autonomous driving capabilities and enhancing accident anticipation. Existing studies primarily predict human scanpath for visual search, visual question answering, and free-viewing. Few studies have focused on predicting scanpath in driving scenarios. To address these limitations, we propose a novel inverse reinforcement learning-based approach through adversarial learning to effectively anticipate human-like scanpaths within different driving tasks. Particularly, we introduce a Transformer-based architecture to construct the generator and discriminator models, while integrating top-down, bottom-up, and historical information for dynamic state updated through State-Encode-with-Attention (SEA). Inspired by the human visual system, SEA adopts a fovea-like movement strategy. Experimental results on the benchmark dataset of BDD-X-diverse validate the effectiveness of our proposed method.

Scanpath refers to the trajectory of eye fixations when humans perform visual reasoning. Most existing methods mainly focus on predicting static attention maps, which represent the probability that each pixel in the image is paid attention to by humans. However, human gaze behavior is purposeful and dynamic, especially in the search for specific objects. Inspired by eye-movement mechanism of human vision system, a reinforcement learning method is introduced to imitate the human visual system to predict scanpath in target search. This paper also considers periphery-fovea vision and incorporates eye-movement behavior to improve the accuracy of scanpath prediction. Besides, the Contrastive Language-Image Pretraining (CLIP) text encoder is employed as the task embedding to convert target objects into vectors. Compared with the state-of-the-art (SOTA) models on COCO-Search18 dataset, our proposed method achieves comprehensively superior performance on fixations location and duration prediction.

Existing investigations on action understanding have made noteworthy advancements by treating activities as holistic events occurring in videos. However, these investigations have limited ability to comprehensively extract and represent human experiential knowledge, which hampers various practical applications, such as robotics and human-computer interaction. We argue that human actions can be better understood as hierarchical compositions of multiple interactive objects and atomic actions with spatio-temporal relations. To this end, we propose a hierarchical understanding framework for home actions, which decomposes a single holistic action into multiple quintuples of. Within this framework, we introduce a two-stage network architecture that leverages multiple prior knowledge in both implicit and explicit ways to facilitate mutual learning within quintuples. In particular, we fully exploit statistical knowledge to enhance the inference of data-driven visual model. Experiments validate the effectiveness of our proposed method. This is also the winning solution for HOMAGE Competition @ ActivityNet Challenge in CVPR 2022 and our brief oral representation is available at https://youtu.be/KK3SPK6iueE?si=hrFZzSABNyrrL6jF&t=1727.

With the increasing popularity of autonomous driving, a demand for high-quality safety-critical driving video data is urgently required. However, such large-scale data is hard to obtain due to expensive and risky collection costs. To alleviate the problem, we propose DrivingGen, an efficient approach built upon the T2I diffusion model for safety-critical driving video generation. Our model employs the "Spatio-Temporal-then-Temporal" paradigm, learning motion priors from a local to global perspective. Firstly, we design an innovative Segment Flow Module to achieve local spatio-temporal modeling by capturing the distinctive dynamic features of different video segments. Secondly, a lightweight Directional Consistency Attention is proposed to further enhance temporal consistency from a global perspective. Additionally, we propose an efficient Temporal Shift Adapter to expand the T2I U-Net into the temporal dimension. Empowered with these modules, DrivingGen outperforms the state-of-the-arts in driving video generation for safety-critical scenarios, as determined by both quality and efficiency measures. Video examples are available in our project page: https://gzp6688.github.io/DrivingGen

Since the inception of Industry 5.0 in 2021, a growing number of researchers have begun to pay their attention to the revolutionary shift it brings. The principles of Industry 5.0, including human-centric, sustainability, and emphasis on ecological and social values, will become the new paradigm for future industrial development. In this transformative landscape, artificial intelligence (AI) plays a pivotal role, and foundation models based on ChatGPT are set to reshape the organizational structure of industries. In this article, we introduce a multimodal perception and decision-making system built upon a foundational model. This system integrates image and point cloud data to enhance perception accuracy and provide ample information for decision making. It is designed to achieve a deep integration of AI and human-centric autonomous driving within the context of Industry 5.0. We introduce a cross-domain learning approach in the system architecture, along with a model training method from foundation models to handle complex road conditions. The proposed method enables road drivable area segmentation on complex unstructured roads. To address the issue of increased variance caused by the residual structure employed in previous works, this article introduces a distribution correction module, which effectively mitigates this problem. Furthermore, to achieve high-performance perception systems in intricate road scenarios, we put forth a multimodal perception fusion method in this study. The experiments demonstrate the superiority of this approach over single-sensor perception. This work contributes to the ongoing discourse on the convergence of AI, human-centric values, and advanced driving systems within the framework of Industry 5.0.

Safety-critical driving scenarios are essential to the development and validation of autonomous driving algorithms. Currently, most of the data is acquired in naturalistic scenarios, resulting in a sparsity of the safety-critical cases. Consequently, synthetic scenario generation based on deep models becomes crucial to validate the risk and reduce the cost. However, previous works either fail to generate realistic scenarios with high quality or can hardly follow instructions to generate desired scenarios. In this paper, we propose a controllable diffusion model that operates on a naturalistic driving scenario to generate safety-critical cases with high fidelity and controllability. In particular, our proposed method encompasses a pre-trained text-to-image diffusion model and a bounding box to incorporate the conditions of category and position of generated objects, respectively. To enhance generation quality and mitigate boundary artifacts, we introduce a mask-aware adapter to better integrate the generated objects into the driving scenarios. Moreover, we propose a transformer encoder and region-guided cross attention to fuse the additional coordinate inputs into the diffusion model, while encouraging more interaction between different generated objects. Comparative analysis demonstrates that our method outperforms exciting work by offering more realistic, diverse and controllable synthetic scenarios and allowing for multiple objects generation with complex spatial relationship.

Human-object interaction (HOI) detection aims to interpret the interactions of human-object pairs. Existing methods adopt a one-step reasoning paradigm that simultaneously outputs multi-label results for all HOI pairs without distinguishing difficulties. However, there are significant variations among HOI pairs in the same image, making their performance degrade in challenging situations. In this paper, we argue that the model should prioritize hard samples after inferring easy ones, and hard samples can benefit from easy ones. To this end, we propose a novel Multi-step Reasoning Network that progressively learns from easy to hard samples. In particular, an Easy-to-Hard Learning Block is introduced to enhance the representation of hard HOI pairs by prior associations. Additionally, we propose a Multi-step Reasoning Probability Transfer mechanism to enhance multi-label interaction classifications, which leverages cognitive associations and semantic dependencies. Extensive experiments demonstrate that our method outperforms other state-of-the-art on two challenging benchmark datasets.

For autonomous driving, one of the major challenges is to predict pedestrian crossing intention in ego-view. Pedestrian intention depends not only on their intrinsic goals but also on the stimulation of surrounding traffic elements. Considering the influence of other traffic elements on pedestrian intention, recent work introduced more traffic element information into the model to successfully improve performance. However, it is still difficult to effectively capture and fully exploit the potential dynamic spatio-temporal interactions among the target pedestrian and its surrounding traffic elements for accurate reasoning. In this work, inspired by neuroscience that human drivers tend to make continuous sensory-motor driving decisions by progressive visual stimulation, we propose a model termed Progressive Interaction Transformer (PIT) for pedestrian crossing intention prediction. Local pedestrian, global environment, and ego-vehicle motion are considered simultaneously in the proposed PIT. In particular, the temporal fusion block and self-attention mechanism are introduced to jointly and progressively model the dynamic spatio-temporal interactions among the three parties, allowing it to capture richer information and make prediction in a similar way to human drivers. Experimental results demonstrate that PIT achieves higher performance compared with other state-of-the-arts and preserves real-time inference.

Eye tracking technology is widely used in a range of potential interactive applications, containing biometrics recognition, emotion recognition, and virtual reality. Eye tracking includes various tasks, but most existing methods cannot accomplish multiple tasks simultaneously. The related works conduct eye localization first, followed by performing gaze estimation or eye state prediction sequentially. In this paper, we propose a unified method based on cascade regression framework to achieve multi-task of eye detection, eye state prediction, and gaze estimation simultaneously. We hypothesize that there is a correspondence between each task, hence introducing a cascade regression framework to capture the implicit relation. At each iteration, we extract appearance features and shape features from eye region to estimate eye state and gaze direction. Based on previous eye state information and gaze vectors, we further use the cascade regression to map these information to update eye location. The proposed method accomplishes three tasks, namely eye state prediction, gaze estimation, and eye localization, through learning the cascade regression. Experimental results on the benchmarks of BioID, Gi4E, MPIIGaze and UT-Multiview demonstrate that the proposed approach achieves superior performance in the aforementioned three tasks.

Public concerns on image encryption grow significantly as the development and application of edge computing and the Internet of Things intensified recently. However, most existing image cryptosystems are not sophisticated enough to resist the two major attack strategies available currently, that is: 1) differential attacks and 2) chosen-plaintext attacks, which are famous for their destructive power, especially their capability of exploiting cryptosystems' features to recover the secret key. In this article, we propose an artificial image, computational experiment, and parallel execution (ACP)-based color image encryption approach using redundant blocks. First, a redundant blocks strategy with redundant spaces is proposed to prevent differential attacks and accelerate operating speed while guaranteeing the security of image cryptosystems. Second, real-world chaotic data (e.g., stock data) are obtained to generate artificial images and conduct computational experiments. Furthermore, artificial images are encrypted via real-world chaos, while the original images are encrypted via simulated chaos (such as Chen's hyperchaos). Finally, we design the process of parallel execution for image encryption and use DNA XOR to merge two groups of encrypted subimages to fuse the effect of the chaotic characteristics in both the real world and the simulation. The final encrypted image is realized through the recovery of redundant blocks. The ACP mechanism of color image encryption achieves the goal of improving the sophistication of chaos-based cryptosystems and resists both the differential and chosen-plaintext attacks. Experimental results and security analysis show that our approach provides not only excellent encryption but also security sufficient to prevent known attacks.

Deep learning is a compelling technique for feature extraction due to its adaptive capacity of processing and providing deeper image information. However, for the task of leaf cultivar classification, the deep learning-based classifier model is unable to extract contour features of leaf images deeply due to the lack of large specialized datasets and expert knowledge annotations. Also, the scale/size of the current leaf cultivar dataset does not meet the needs of deep neural networks (DNNs). In particular, the high model complexity of DNNs implies that deep-learning-based neural networks seem to must require a large dataset to achieve good performance, but facing the fact that the leaf cultivar dataset often is small, even some classes in this kind of datasets contain less than ten images/examples. To overcome these problems and inspired by the resounding success of fuzzy logic, we propose a novel fuzzy ensemble model for leaf cultivar classification. To extract the contours of leaves, we first propose generative adversarial networks-based methods. Second, to improve the ability of feature representation, we present a data augmentation method to transform our contour features. Third, to get the essential features of leaves, we design a novel generation of thefuzzy random forest. Finally, to achieve accurate classification, we design a novel deep learning strategy, namelydeep fuzzy representation learning, integrating and cascading a lot of our fuzzy random forests. Experimental results show that our model outperforms other existing state-of-the-arts on three real-world datasets, and performs much better than the original deep forest and DNN-based algorithms particularly.

Cellular signaling data (CSD) have attracted unprecedented attention due to their large size, long observation period, and high followability. Before applying CSD, a series of data processing steps are indispensable; among those steps, staying point recognition is the basis for recognizing individual travel states and thus the influence of further application of CSD. Previous work indicates that the existing staying point recognition algorithms have two common aspects. One is the requirement of a fixed spatiotemporal threshold to analyze the user’s travel characteristics. The other is the insufficiency of accuracy assessment, which indicates that further studies are expected owing to the lack of ground truth data in CSD. In this work, a “spatiotemporal window”-based algorithm is proposed to recognize individual staying and moving states. First, an iterative-learning-based model is designed to cluster individual trajectory points without predefined spatiotemporal thresholds. Then, rules to distinguish the staying or moving cluster are made from individual travel characteristics. Moreover, verification work is carried out by collecting volunteers’ ground truth data using our developed smartphone application, which achieves an accuracy of 91.3%. Finally, the results demonstrate the effectiveness and robustness of the algorithm through the performance of comparison and sensitivity analyses.

Autonomous driving has become a prevalent research topic in recent years, arousing the attention of many academic universities and commercial companies. As human drivers rely on visual information to discern road conditions and make driving decisions, autonomous driving calls for vision systems such as vehicle detection models. These vision models require a large amount of labeled data while collecting and annotating the real traffic data are time‐consuming and costly. Therefore, we present a novel vehicle detection framework based on the parallel vision to tackle the above issue, using the specially designed virtual data to help train the vehicle detection model. We also propose a method to construct large‐scale artificial scenes and generate the virtual data for the vision‐based autonomous driving schemes. Experimental results verify the effectiveness of our proposed framework, demonstrating that the combination of virtual and real data has better performance for training the vehicle detection model than the only use of real data.

Even with a still image, humans can ratiocinate various visual cause-and-effect descriptions before, at present, and after, as well as beyond the given image. However, it is challenging for models to achieve such task–the visual event ratiocination, owing to the limitations of time and space. To this end, we propose a novel multi-modal model, Hypergraph-Enhanced Graph Reasoning. First it represents the contents from the same modality as a semantic graph and mines the intra-modality relationship, therefore breaking the limitations in the spatial domain. Then, we introduce the Graph Self-Attention Enhancement. On the one hand, this enables semantic graph representations from different modalities to enhance each other and captures the inter-modality relationship along the line. On the other hand, it utilizes our built multi-modal hypergraphs in different moments to boost individual semantic graph representations, and breaks the limitations in the temporal domain. Our method illustrates the case of "two heads are better than one" in the sense that semantic graph representations with the help of the proposed enhancement mechanism are more robust than those without. Finally, we re-project these representations and leverage their outcomes to generate textual cause-and-effect descriptions. Experimental results show that our model achieves significantly higher performance in comparison with other state-of-the-arts.

This paper deals with the challenging problem of building visual reasoning models for answering questions related to artworks in artistic domains. The nature of abstract styles and cultural contexts within an artistic image makes the corresponding learning tasks extremely difficult. We propose a novel framework termed as Hierarchical-Knowledge Embedded Meta-Learning to address the critical issues of visual reasoning in artistic domains. In particular, we firstly present a deep relational model to capture and memorize the relations among different samples. Then, we provide the hierarchical-knowledge embedding that mines the implicit relationship between question-answer pairs for knowledge representation as the guidance of our meta-learner. This is a case of "knowledge is power" in the sense that the hierarchical knowledge representation is incorporated into our meta-learning based model. The final classification is derived from our model by learning to compare the features of samples. Experimental results show that our approach achieves significantly higher performance compared with other state-of-the-arts.

Person search often requires a query photo of the target person. However, in many practical scenarios, there is no guarantee that such a photo is always available. In this paper, we define the problem of sketch based person search, which uses a sketch instead of a photo as the probe for retrieving. We tackle this problem in a weak supervision setting and propose a clustering and feature attention based weakly supervised learning framework, which contains two stages of pedestrian detection and sketch based person re-identification. Specially, we introduce multiple detectors, followed by fuzzy c-means clustering to achieve weakly supervised pedestrian detection. Moreover, we design an attention module to learn discriminative features in subsequent re-identification network. Extensive experiments show the superiority of our method.

Adversarial images generated by generative adversarial networks are not close to any existing benign images, and contain nonrobust features that have been identified as critical to the robustness of a machine learning model. Since adversarial images have an underlying distribution that differs from normal images, these kinds of images can offer valuable features for training a robust model. To deal with these special features, we focus on a novel machine learning task of adversarial images classification, where adversarial images can be used to investigate the problem of classifying adversarial images themselves. In the setting of this novel task, adversarial images are the ONLY kind of data used in training and testing, rather than not just a set of testing images as usual. To this end, we propose a novel spatial–frequency ensemble relation network with generative adversarial learning. First, we present a spatial–frequency ensemble representation learning to extract the feature of training images. Second, we design a meta-learning-based relation model to gain the relationship between images. Third, to achieve a robust model, we utilize generative adversarial learning and transform the relationship into a Jacobian matrix. Finally, we design a discriminator model that determines whether an adversarial image is from the matching category or not. Experimental results demonstrate that our approach achieves significantly higher performance compared with other state-of-the-arts.

The goal of diagnosing the coronavirus disease 2019 (COVID-19) from suspected pneumonia cases, that is, recognizing COVID-19 from chest X-ray or computed tomography (CT) images, is to improve diagnostic accuracy, leading to faster intervention. The most important and challenging problem here is to design an effective and robust diagnosis model. To this end, there are three challenges to overcome: (1) The lack of training samples limits the success of existing deep-learning-based methods. (2) Many public COVID-19 data sets contain only a few images without fine-grained labels. (3) Due to the explosive growth of suspected cases, it is urgent and important to diagnose not only COVID-19 cases but also the cases of other types of pneumonia that are similar to the symptoms of COVID-19. To address these issues, we propose a novel framework called Unsupervised Meta-Learning with Self-Knowledge Distillation to address the problem of differentiating COVID-19 from pneumonia cases. During training, our model cannot use any true labels and aims to gain the ability of learning to learn by itself. In particular, we first present a deep diagnosis model based on a relation network to capture and memorize the relation among different images. Second, to enhance the performance of our model, we design a self-knowledge distillation mechanism that distills knowledge within our model itself. Our network is divided into several parts, and the knowledge in the deeper parts is squeezed into the shallow ones. The final results are derived from our model by learning to compare the features of images. Experimental results demonstrate that our approach achieves significantly higher performance than other state-of-the-art methods. Moreover, we construct a new COVID-19 pneumonia data set based on text mining, consisting of 2696 COVID-19 images (347 X-ray + 2349 CT), 10,155 images (9661 X-ray + 494 CT) about other types of pneumonia, and the fine-grained labels of all. Our data set considers not only a bacterial infection or viral infection which causes pneumonia but also a viral infection derived from the influenza virus or coronavirus.

Visual reasoning between visual image and natural language description is a long-standing challenge in computer vision. While recent approaches offer a great promise by compositionality or relational computing, most of them are oppressed by the challenge of training with datasets containing only a limited number of images with ground-truth texts. Besides, it is extremely time-consuming and difficult to build a larger dataset by annotating millions of images with text descriptions that may very likely lead to a biased model. Inspired by the majority success of webly supervised learning, we utilize readily-available web images with its noisy annotations for learning a robust representation. Our key idea is to presume on web images and corresponding tags along with fully annotated datasets in learning with knowledge embedding. We present a two-stage approach for the task that can augment knowledge through an effective embedding model with weakly supervised web data. This approach learns not only knowledge-based embeddings derived from key-value memory networks to make joint and full use of textual and visual information but also exploits the knowledge to improve the performance with knowledge-based representation learning for applying other general reasoning tasks. Experimental results on two benchmarks show that the proposed approach significantly improves performance compared with the state-of-the-art methods and guarantees the robustness of our model against visual reasoning tasks and other reasoning tasks.

While existing person search methods have achieved good performance, they require the images used for training contain labels about the identity and bounding box location of each person. However, it is expensive and difficult to manually annotate these labels in the large scale scenario. To overcome this issue, we consider weakly supervised person search. The weakly supervised setting means during training we only know which identities appear in the image set and how many individuals present in each image, without any identity or location information on the image. Facing this challenge, we propose a clustering and patch based weakly supervised learning (CPBWSL) framework, which separately addresses two sub-tasks including pedestrian detection and person re-identification. Particularly, we introduce multiple detectors to provide more detection results as well as fuzzy c-means clustering algorithm to cluster these results and remove low membership ones. Moreover, a patch based learning network is designed to generate different patches and learn discriminative patch features. Extensive experiments on two benchmarks indicate that the proposed weakly supervised setting is feasible and our method can achieve performance comparable to some fully supervised person search methods.

Image-to-image translation aims to learn the mapping between two visual domains. At the beginning of designing the existing image-to-image translation method, it was not considered whether the generated image is realistic or not. In this work, we present a novel approach to address the problem of generating fidelity in the area of image-to-image translation. In particular, humans judge whether an image is realistic or not with unique human vision's feeling rather than paying attention to the real-world semantics. Inspired by this, we propose an effective network loss to capture the pixel-level representations and human vision system information for verisimilar image-to-image translation. To enforce both structural and translation-model consistency during adaptation, we propose a novel Just-Noticeable-Difference loss based on a visual recognition task. The Just-Noticeable-Difference loss not only guides the overall representation to be discriminative, but also enforces our cycle loss before and after mapping between domains. Qualitative results show that our model can generate realistic images on a wide range of tasks without paired training data. For quantitative comparisons, we measure realism with user study and diversity with a perceptual distance metric. We apply the proposed model to domain adaptation and show competitive performance when compared to the state-of-the-art on many datasets.

Due to its ill-posed nature, single image dehazing is a challenging problem. In this paper, we propose an end-to-end feature aggregation attention network (FAAN) for single image dehazing. It incorporates the idea of attention mechanism and residual learning and can adaptively aggregate different level features. In particular, in the proposed FANN, we design a novel block structure consisting of feature attention module, smoothed dilated convolution and local residual learning. The local residual learning allows the less useful information to be bypassed through multiple skip connections. The feature attention module is designed to assign more weight to important features. The smoothed dilated convolution is adopted to enlarge the receptive field without the negative influence of gridding artifacts. The experiments on the RESIDE dataset show that the proposed approach acquires state-of-the-art performance in both qualitative and quantitative measures.

With more than 300 million people depressed worldwide annually, depression is a global problem. The goal of depression detection is to improve diagnostic accuracy and availability, leading to faster intervention. The most important and challenging problem here is to design an effective and robust depression detection model. To this end, there are two challenges to overcome: 1) Multi-modal (audio, image, text, etc.) information must be jointly considered to make accurate inferences. 2) Existing deep learning-based work suffers from multi-modal data sufficiency problem. To address these issues, we propose a graph attention model embedded with multi-modal knowledge for depression detection. This approach learns not only reasonable embeddings for nodes in the knowledge graph, but also exploits medical knowledge to improve the performance of classification and prediction with the knowledge attention mechanism. Experimental results on the real-world datasets show that the proposed approach significantly improves the classification and prediction performance compared with other major state-of-the-art approaches, with guaranteed the robustness with each modality of multi-modal data. Overall, this paper shows how multi-modal knowledge attention mechanism and deep-learning-based networks can be combined to assist mental health patients and practitioners.

In this paper, we propose a feature selection method that characterizes the difference between two kinds of probability distributions. The key idea is to view the feature selection problem as a sparsest k-subgraph problem that considers Wasserstein distance between the studied two probability distributions. Our method does not presume any specific parametric models on the data distribution and is non-parametric. It outperforms existing Kullback-Leibler divergence based approaches, since we do not require two distributions to overlap. This relaxation makes our method work in many problems in which Kullback-Leibler divergence based methods fail. We also design a fast calculation algorithm using dynamic programming. Our experimental results show that our method outperforms the current method in both computation accuracy and speed.

Credit card transaction fraud costs billions of dollars to card issuers every year. Besides, the credit card transaction dataset is very skewed, there are much fewer samples of frauds than legitimate transactions. Due to the data security and privacy, different banks are usually not allowed to share their transaction datasets. These problems make traditional model difficult to learn the patterns of frauds and also difficult to detect them. In this paper, we introduce a novel framework termed as federated meta-learning for fraud detection. Different from the traditional technologies trained with data centralized in the cloud, our model enables banks to learn fraud detection model with the training data distributed on their own local database. A shared whole model is constructed by aggregating locallycomputed updates of fraud detection model. Banks can collectively reap the benefits of shared model without sharing the dataset and protect the sensitive information of cardholders. To achieve the good performance of classification, we further formulate an improved triplet-like metric learning, and design a novel meta-learning-based classifier, which allows joint comparison with K negative samples in each mini-batch. Experimental results demonstrate that the proposed approach achieves significantly higher performance compared with the other state-of-the-art approaches.

This paper deals with a challenging task of learning from different modalities by tackling the difficulty problem of jointly face recognition between abstract-like sketches, cartoons, caricatures and real-life photographs. Due to the significant variations in the abstract faces, building vision models for recognizing data from these modalities is an extremely challenging. We propose a novel framework termed as Meta-Continual Learning with Knowledge Embedding to address the task of jointly sketch, cartoon, and caricature face recognition. In particular, we firstly present a deep relational network to capture and memorize the relation among different samples. Secondly, we present the construction of our knowledge graph that relates image with the label as the guidance of our meta-learner. We then design a knowledge embedding mechanism to incorporate the knowledge representation into our network. Thirdly, to mitigate catastrophic forgetting, we use a meta-continual model that updates our ensemble model and improves its prediction accuracy. With this meta-continual model, our network can learn from its past. The final classification is derived from our network by learning to compare the features of samples. Experimental results demonstrate that our approach achieves significantly higher performance compared with other state-of-the-art approaches.

As the size and source of network traffic increase, so does the challenge of monitoring and analyzing network traffic. The challenging problems of classifying encrypted traffic are the imbalanced property of network data, the generalization on an unseen dataset, and overly dependent on data size. In this paper, we propose an application of a meta-learning approach to address these problems in encrypted traffic classification, named Flow-Based Relation Network (RBRN). The RBRN is an end-to-end classification model that learns representative features from the raw flows and then classifies them in a unified framework. Moreover, we design “hallucinator” to produce additional training samples for the imbalanced classification, and then focus on meta-learning to classify unseen categories from few labeled samples. We validate the effectiveness of the RBRN on the real-world network traffic dataset, and the experimental results demonstrate that the RBRN can achieve an excellent classification performance and outperform the state-of-the-art methods on encrypted traffic classification. What is more interesting, our model trained on the real-world dataset can generalize very well to unseen datasets, outperforming multiple state-of-art methods.

Automatic diagnosis based on medical imaging necessitates both lesion segmentation and disease classification. Lesion segmentation requires pixel-level annotations while disease classification only requires image-level annotations. The two tasks are usually studied separately despite the latter problem relies on the former. Motivated by the close correlation between them, we propose a mixed-supervision guided method and a residual-aided classification U-Net model (ResCU-Net) for joint segmentation and benign-malignant classification. By coupling the strong supervision in the form of segmentation mask and weak supervision in the form of benign-malignant label through a simple annotation procedure, our method efficiently segments tumor regions while simultaneously predicting a discriminative map for identifying the benign-malignant types of tumors. Our network, ResCU-Net, extends U-Net by incorporating the residual module and the SegNet architecture to exploit multilevel information for achieving improved tissue identification. With experiments on a public mammogram database of INbreast, we validate the effectiveness of our method and achieve consistent improvements over state-of-the-art models.

Breast cancer diagnosis based on medical imaging necessitates both fine-grained lesion segmentation and disease grading. Although deep learning (DL) offers an emerging and powerful paradigm of feature learning for these two tasks, it is hampered from popularizing in practical application due to the lack of interpretability, generalization ability, and large labeled training sets. In this article, we propose a hierarchical fused model based on DL and fuzzy learning to overcome the drawbacks for pixelwise segmentation and disease grading of mammography breast images. The proposed system consists of a segmentation model (ResU-segNet) and a hierarchical fuzzy classifier (HFC) that is a fusion of interval type-2 possibilistic fuzzy c-means and fuzzy neural network. The ResU-segNet segments the masks of mass regions from the images through convolutional neural networks, while the HFC encodes the features from mass images and masks to obtain the disease grading through fuzzy representation and rule-based learning. Through the integration of feature extraction aided by domain knowledge and fuzzy learning, the system achieves favorable performance in a few-shot learning manner, and the deterioration of cross-dataset generalization ability is alleviated. In addition, the interpretability is further enhanced. The effectiveness of the proposed system is analyzed on the publicly available mammogram database of INbreast and a private database through cross-validation. Thorough comparative experiments are also conducted and demonstrated.

Automated plant species identification system could help botanists and layman in identifying plant species rapidly. Deep learning is robust for feature extraction as it is superior in providing deeper information of images. However, deep neural networks based leaf cultivar classification model can not deeply mine the contour features of leaf images. Besides, the deep neural networks are very data-hungry due to the large complexity of the models, which means that the model's performance can decrease significantly when the size of the training data decreases. In this paper, we propose the leaf cultivar classification model based on forest representation learning and multiscale contour feature learning. Firstly, we build the multiscale contour feature learning to extract the contour features of leaf images. Then, the original input features are transformed by the data augmentation method to enhance the ability of feature expression, and the autoencoder by forest performs the data of dimensionality reduction on the features. Finally, we use the improved deep forest algorithm and autoencoder by forest to build forest representation learning for classification. The experimental results show that the proposed algorithm has higher performance than the original deep forest (gcForest) algorithm and other existing start-of-art algorithms, and has higher performance and efficiency than other deep learning algorithms.

Face super-resolution has been studied for decades, and many approaches have been proposed to upsample low-resolution face images using information mined from paired low-resolution (LR) images and high-resolution (HR) images. However, most of this kind of works only simply sharpen the blurry edges in the upsampled face images and typically no photo-realistic face is reconstructed in the final result. In this paper, we present a GAN-based algorithm for face super-resolution which properly synthesizes photo-realistic super-recovered face. To this end, we introduce semi-dual optimal transport to optimize our model such that the distribution of its generated data can match the distribution of a target domain as much as possible. This way would endow our model with learning the mapping of distribution from unpaired LR images and HR images with desired properties. We demonstrate the robustness of our algorithm by testing it on Color FERET database and show that its performance is considerably superior to all state-of-the-art approaches.

Caricature is a simple and abstract description of a person using her/his exaggerated characteristics. Due to amplified facial variations in the caricatures and significant differences among caricature and real face modalities, building vision models for recognizing each other between these modalities is an extremely challenging task. In addition, it is not easy to collect abundant samples of real faces and corresponding caricatures for training vision models, which makes the recognition more difficult. In this paper, we propose a novel relation network via meta learning to address the problem of few-shot caricature face recognition. In particular, we present a deep relation network to capture and memorize the relation among different samples. To employ the prior knowledge, we combine learned deep and handcrafted features to form the hybrid-prior representation via joint meta learning. Final recognition is derived from our relation network by learning to compare between the hybrid-prior features of samples. Experimental results on three caricature datasets of WebCaricature, IIIT-CFW, and Caricature-207 demonstrate that our method performs better than many existing ones for few-shot caricature recognition.

Image based eye detection and gaze estimation have a wide range of potential applications, such as medical treatment, biometrics recognition, human-computer interaction. Though a large number of researchers have attempted to solve the two problems, they still exist some challenges due to the variation in appearance and lack of annotated images. In addition, most related work perform eye detection first, followed by gaze estimation via appearance learning. In this paper, we propose a unified framework to execute the gaze estimation and the eye detection simultaneously by learning the cascade regression models from appearance around the eye related key points. Intuitively, there is coupled relationship among location of eye center, shape of eye related key points, appearance representation and gaze information. To incorporate these information, at each cascade level, we first learn a model to map the shape and appearance around current eye related key points to the three dimension gaze update. Then, with the help of estimated gaze, we further learn a regression model to map the gaze, shape and appearance information to eye location update. By leveraging the power of cascade learning, the proposed method can alternatively optimize the two tasks of eye detection and gaze estimation. The experiments are conducted on benchmarks of GI4E and MPIIGaze. Experimental results show that our proposed method can achieve preferable results in gaze estimation and outperform the state-of-the-art methods in eye detection.

Facial landmark detection, head pose estimation, and facial deformation analysis are typical facial behavior analysis tasks in computer vision. The existing methods usually perform each task independently and sequentially, ignoring their interactions. To tackle this problem, we propose a unified framework for simultaneous facial landmark detection, head pose estimation, and facial deformation analysis, and the proposed model is robust to facial occlusion. Following a cascade procedure augmented with model-based head pose estimation, we iteratively update the facial landmark locations, facial occlusion, head pose and facial deformation until convergence. The experimental results on benchmark databases demonstrate the effectiveness of the proposed method for simultaneous facial landmark detection, head pose and facial deformation estimation, even if the images are under facial occlusion.

Current approaches for facial landmark detection and head pose estimation first perform landmark detection, followed by fitting 3D face model or regression model to estimate head pose. Different from the existing methods, in this paper, we propose a unified method, called Coupled Cascade Regression (CCR), for simultaneous facial landmark detection and head pose estimation. At each cascade level, two separate regressors are learned to update the landmark locations and 3D face model parameters based on the local appearance features, respectively. Since 2D facial landmark locations and head pose parameters are related, we further apply the projection model to refine the prediction results in each cascade iteration and make them consistent. As a result, CCR can leverage both the learning methods and the projection model to simultaneously perform facial landmark detection and pose estimation to enhance the performances of both tasks. Experimental results on 300-W and BU datasets indicate that our proposed CCR method outperforms many conventional methods both for landmark detection and head pose estimation.

Traffic surveillance has become an important topic in intelligent transportation systems (ITSs), which is aimed at monitoring and managing traffic flow. With the progress in computer vision, video-based surveillance systems have made great advances on traffic surveillance in ITSs. However, the performance of most existing surveillance systems is susceptible to challenging complex traffic scenes (e.g., object occlusion, pose variation, and cluttered background). Moreover, existing related research is mainly on a single video sensor node, which is incapable of addressing the surveillance of traffic road networks. Accordingly, we present a review of the literature on the video-based vehicle surveillance systems in ITSs. We analyze the existing challenges in video-based surveillance systems for the vehicle and present a general architecture for video surveillance systems, i.e., the hierarchical and networked vehicle surveillance, to survey the different existing and potential techniques. Then, different methods are reviewed and discussed with respect to each module. Applications and future developments are discussed to provide future needs of ITS services.

Cascade regression framework has been successfully applied to facial landmark detection and achieves state-of-the-art performance recently. It requires large number of facial images with labeled landmarks for training regression models. We propose to use cascade regression framework to detect eye center by capturing its contextual and shape information of other related eye landmarks. While for eye detection, it is time-consuming to collect large scale training data and it also can be unreliable for accurate manual annotation of eye related landmarks. In addition, it is difficult to collect enough training data to cover various illuminations, subjects with different head poses and gaze directions. To tackle this problem, we propose to learn cascade regression models from synthetic photorealistic data. In our proposed approach, eye region is coarsely localized by a facial landmark detection method first. Then we learn the cascade regression models iteratively to predict the eye shape updates based on local appearance and shape features. Experimental results on benchmark databases such as BioID and GI4E show that our proposed cascade regression models learned from synthetic data can accurately localize the eye center. Comparisons with existing methods also demonstrates our proposed framework can achieve preferable performance against state-of-the-art methods.

This paper presents a vehicle license plate recognition method based on character-specific extremal regions (ERs) and hybrid discriminative restricted Boltzmann machines (HDRBMs). First, coarse license plate detection (LPD) is performed by top-hat transformation, vertical edge detection, morphological operations, and various validations. Then, character-specific ERs are extracted as character regions in license plate candidates. Followed by suitable selection of ERs, the segmentation of characters and coarse-to-fine LPD are achieved simultaneously. Finally, an offline trained pattern classifier of HDRBM is applied to recognize the characters. The proposed method is robust to illumination changes and weather conditions during 24 h or one day. Experimental results on thorough data sets are reported to demonstrate the effectiveness of the proposed approach in complex traffic environments.

Traffic surveillance has become an important topic in intelligent transportation systems (ITSs), which is aimed at monitoring and managing traffic flow. With the progress in computer vision, video-based surveillance systems have made great advances on traffic surveillance in ITSs. However, the performance of most existing surveillance systems is susceptible to challenging complex traffic scenes (e.g., object occlusion, pose variation, and cluttered background). Moreover, existing related research is mainly on a single video sensor node, which is incapable of addressing the surveillance of traffic road networks. Accordingly, we present a review of the literature on the video-based vehicle surveillance systems in ITSs. We analyze the existing challenges in video-based surveillance systems for the vehicle and present a general architecture for video surveillance systems, i.e., the hierarchical and networked vehicle surveillance, to survey the different existing and potential techniques. Then, different methods are reviewed and discussed with respect to each module. Applications and future developments are discussed to provide future needs of ITS services.