Many real-world problems involve optimizing numerous decision variables and are expensive to evaluate, known as large-scale expensive optimization problems (LSEOPs). While surrogate-assisted evolutionary algorithms have proven effective for expensive problems, training proper models for LSEOPs remains challenging due to insufficient training data. In this paper, we adopt the divide-and-conquer approach, decomposing LSEOPs into lower-dimensional sub-problems and constructing models for sub-problems, and introduce a multi-view synthetic sampling technique for new sample selection. Specifically, we propose sorting all evaluated solutions in an ascending order and dividing them into intervals, from which data are sampled to obtain informative training data for models. The population for the LSEOP is updated by employing cooperative environmental selections on the population, formed by recombining all renewed populations for sub-problems to balance exploration and exploitation. Finally, a solution is selected among the current population for the true evaluation based on its multi-view performance predicted across all sub-problems. Results on CEC’2013 benchmark problems show the effectiveness and efficiency of our proposed method compared to three prevalent large-scale expensive optimization algorithms. Additionally, results on 2000-dimensional CEC’2010 benchmark problems and a 1200-dimensional real-world problem demonstrate encouraging scalability and robustness of the proposed method for addressing higher-dimensional problems.

Traditional large-scale evolutionary algorithms are limited in their ability to solve certain real-world applications with high-dimensional, closed-box, and computationally expensive objectives due to their need for numerous objective evaluations. Surrogate-assisted evolutionary algorithms (SAEAs) have shown effective for expensive closed-box optimization by relying on inexpensive surrogate models. However, large-scale optimization remains challenging for SAEAs due to the exponentially growing search space and the presence of multiple local optima, resulting in difficulty in training a proper model due to the lack of samples. To address these challenges, we propose constructing an initial surrogate model on randomly selected dimensions and calculating a Gaussian distribution for each sampled dimension. The surrogate then provides predictions when perturbing each sampled dimension by sampling from the distribution, enabling the identification of the most important variables for constructing an active subproblem to reduce the search space. A secondary surrogate model, built for the active subproblem, guides the offspring generation and environmental selection for a modified particle swarm optimization algorithm to effectively explores the subspace while escaping local optima in large-scale problems. Experimental results on CEC’2013 and CEC’2010 benchmark problems show that the proposed method outperforms state-of-the-art algorithms in addressing large-scale expensive optimization problems. The efficiency of the proposed method is further verified on CEC’2010 benchmark problems extended to 2000 dimensions.

Large-scale expensive optimization problems (LSEOPs) are characterized by a large number of dimensions in the decision space and expensive objective functions. The curse of dimensionality poses challenges to achieving the optimal solution, especially when limited objective evaluations are allowed. In this paper, the cooperative coevolution framework is used. A PSO variant is proposed to update solutions for each sub-problem. Then, a difference-based infill criterion is proposed to select a solution for real expensive objective evaluation. A number of experiments are conducted on CEC'2013 test problems to verify the performance of the proposed method. The experimental results show that our proposed method can obtain better performance than the four representative methods presented for solving LSEOPs.

Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as 'Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense features.To integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during training.Unlike existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student model. We conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models.

Differentiable neural architecture search (NAS) methods have achieved comparable results for low search costs and high performance. Existing differentiable methods focus on searching microstructures in micro space, lacking in exploring macrostructures. However, different networks should have different macrostructures rather than all evenly distributed. This article proposes an M2NAS optimization system to optimize macrostructure and microstructure jointly. Specifically, we initialize a network with maximum complexity, where down-samplings occur at the beginning. Then, iteratively optimize the macrostructure and microstructure. For macro search, we establish a macro space and explore this space by generating candidates and initializing weights and architectural parameters for candidate networks, i.e., network transmission. For micro search, we propose a progressive pruning method to eliminate the vast quantization error caused by one-time pruning. With the number of parameters decreasing during the search, M2NAS obtains a series of networks with different complexity through network selection, forming a Pareto-optimal set. Results show that networks obtained by M2NAS have a better tradeoff between accuracy and complexity. A network searched on ImageNet achieves 77.4% and 76.3% top-1 accuracy, with and without data augment during retraining. When taking different pretrained models as backbones and combining them with Faster RCNN on COCO, our model can get 35.6%$(AP)$and 55.7%$(AP_{50})$, higher than typical NAS models, second only to ResNet-50 results.

Most methods tackle zero-shot video classification by aligning visual-semantic representations within seen classes, which limits generalization to unseen classes. To enhance model generalizability, this paper presents an end-to-end framework that preserves alignment and uniformity properties for representations on both seen and unseen classes. Specifically, we formulate a supervised contrastive loss to simultaneously align visual-semantic features (i.e., alignment) and encourage the learned features to distribute uniformly (i.e., uniformity). Unlike existing methods that only consider the alignment, we propose uniformity to preserve maximal-info of existing features, which improves the probability that unobserved features fall around observed data. Further, we synthesize features of unseen classes by proposing a class generator that interpolates and extrapolates the features of seen classes. Besides, we introduce two metrics, closeness and dispersion, to quantify the two properties and serve as new measurements of model generalirability. Experiments show that our method significantly outperforms SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51. Code is available11https://github.com/ShipuLoveMili/CVPR2022-AURL.

Curvilinear structure segmentation (CSS) is under semantic segmentation, whose applications include crack detection, aerial road extraction, and biomedical image segmentation. In general, geometric topology and pixel-wise features are two critical aspects of CSS. However, most semantic segmentation methods only focus on enhancing feature representations while existing CSS techniques emphasize preserving topology alone. In this paper, we present a Joint Topology-preserving and Feature-refinement Network (JTFN) that jointly models global topology and refined features based on an iterative feedback learning strategy. Specifically, we explore the structure of objects to help preserve corresponding topologies of predicted masks, thus design a reciprocative two-stream module for CSS and boundary detection. In addition, we introduce such topology-aware predictions as feedback guidance that refines attentive features by supplementing and enhancing saliencies. To the best of our knowledge, this is the first work that jointly addresses topology preserving and feature refinement for CSS. We evaluate JTFN on four datasets of diverse applications: Crack500, CrackTree200, Roads, and DRIVE. Results show that JTFN performs best in comparison with alternative methods. Code is available.1

The competitive performances in visual tracking are achieved mostly by tracking-by-detection based approaches, whose accuracy highly relies on a binary classifier that distinguishes targets from distractors in a set of candidates. However, severe class imbalance, with few positives (e.g., targets) relative to negatives (e.g., backgrounds), leads to degrade accuracy of classification or increase bias of tracking. In this paper, we propose an imbalance-elimination mechanism, which adopts a multi-class paradigm and utilizes a novel candidate generation strategy. Specifically, our multi-class model assigns samples into one positive class and four proposed negative classes, naturally alleviating class imbalance. We define negative classes by introducing proportions of targets in samples, which values explicitly reveal relative scales between targets and backgrounds. Further-more, during candidate generation, we exploit such scale-aware negative patterns to help adjust searching areas of candidates to incorporate larger target proportions, thus more accurate target candidates are obtained and more positive samples are included to ease class imbalance simultaneously. Extensive experiments on standard benchmarks show that our tracker achieves favorable performance against the state-of-the-art approaches, and offers robust discrimination of positive targets and negative patterns.

Visual tracking is the task of estimating the trajectory of a target given its first location in a video. Existing tracking-by-detection approaches build trackers on binary classifiers. However, these approaches are sometimes not discriminative enough to distinguish the target from the background, which impedes the performance under challenging conditions. In this paper, we demonstrate that conventional initializing strategies make the model insufficiently trained. To make full use of the information contained in the first frame, we propose an enhanced scheme of model initialization. In the scheme, several kinds of negative samples are heuristically defined and a multi-stage learning mechanism is adopted to make the model initialization more efficient and stable. Validation experiment demonstrates the effectiveness of the mechanism. Experiments on benchmark datasets show that the proposed method improves the results of the baseline work and achieves state-of-the-art performance.

We present a scalable weakly supervised clustering approach to learn facial action units (AUs) from large, freely available web images. Unlike most existing methods (e.g., CNNs) that rely on fully annotated data, our method exploits web images with inaccurate annotations. Specifically, we derive a weakly-supervised spectral algorithm that learns an embedding space to couple image appearance and semantics. The algorithm has efficient gradient update, and scales up to large quantities of images with a stochastic extension. With the learned embedding space, we adopt rank-order clustering to identify groups of visually and semantically similar images, and re-annotate these groups for training AU classifiers. Evaluation on the 1 millon EmotioNet dataset demonstrates the effectiveness of our approach: (1) our learned annotations reach on average 91.3% agreement with human annotations on 7 common AUs, (2) classifiers trained with re-annotated images perform comparably to, sometimes even better than, its supervised CNN-based counterpart, and (3) our method offers intuitive outlier/noise pruning instead of forcing one annotation to every image. Code is available.

Region learning (RL) and multi-label learning (ML) have recently attracted increasing attentions in the field of facial Action Unit (AU) detection. Knowing that AUs are active on sparse facial regions, RL aims to identify these regions for a better specificity. On the other hand, a strong statistical evidence of AU correlations suggests that ML is a natural way to model the detection task. In this paper, we propose Deep Region and Multi-label Learning (DRML), a unified deep network that simultaneously addresses these two problems. One crucial aspect in DRML is a novel region layer that uses feed-forward functions to induce important facial regions, forcing the learned weights to capture structural information of the face. Our region layer serves as an alternative design between locally connected layers (i.e., confined kernels to individual pixels) and conventional convolution layers (i.e., shared kernels across an entire image). Unlike previous studies that solve RL and ML alternately, DRML by construction addresses both problems, allowing the two seemingly irrelevant problems to interact more directly. The complete network is end-to-end trainable, and automatically learns representations robust to variations inherent within a local region. Experiments on BP4D and DISFA benchmarks show that DRML performs the highest average F1-score and AUC within and across datasets in comparison with alternative methods.

Tracking-by-detection with temporal smoothness has recently attracted increasing attentions in the field of multi-object tracking. Occlusions and clutter are two key problems. To address these problems, this paper proposes a new structure and appearance preserving network flow (SAPNF) with tracking-by-detection, introducing spatial structural configuration and appearance overlapping constraint from frame to frame. One crucial aspect in SAPNF is to consider structure information and appearance smoothness simultaneously which benefits from each other. Unlike previous studies that only learn spatial information or appearance smoothness, a unified min-cost flow with the proposed new structure and appearance induces to track multi-object in crowded and cluttered scenes. Experiments on PETS and TUD benchmarks show that SAPNF performs the comparative results in comparison with alternative methods.

Most action unit (AU) detection methods use one-versus-all classifiers without considering dependences between features or AUs. In this paper, we introduce a joint patch and multi-label learning (JPML) framework that models the structured joint dependence behind features, AUs, and their interplay. In particular, JPML leverages group sparsity to identify important facial patches, and learns a multi-label classifier constrained by the likelihood of co-occurring AUs. To describe such likelihood, we derive two AU relations, positive correlation and negative competition, by statistically analyzing more than 350,000 video frames annotated with multiple AUs. To the best of our knowledge, this is the first work that jointly addresses patch learning and multi-label learning for AU detection. In addition, we show that JPML can be extended to recognize holistic expressions by learning common and specific patches, which afford a more compact representation than the standard expression recognition methods. We evaluate JPML on three benchmark datasets CK+, BP4D, and GFT, using within-and cross-dataset scenarios. In four of five experiments, JPML achieved the highest averaged F1 scores in comparison with baseline and alternative methods that use either patch learning or multi-label learning alone.

The face is one of the most powerful channel of nonverbal communication. The most commonly used taxonomy to describe facial behaviour is the Facial Action Coding System (FACS). FACS segments the visible effects of facial muscle activation into 30+ action units (AUs). AUs, which may occur alone and in thousands of combinations, can describe nearly all-possible facial expressions. Most existing methods for automatic AU detection treat the problem using one-vs-all classifiers and fail to exploit dependencies among AU and facial features. We introduce joint-patch and multi-label learning (JPML) to address these issues. JPML leverages group sparsity by selecting a sparse subset of facial patches while learning a multi-label classifier. In four of five comparisons on three diverse datasets, CK+, GFT, and BP4D, JPML produced the highest average F1 scores in comparison with state-of-the art.

In the realm of facial expression analysis, numerous attempts have been made to link each facial picture to one affective category. Nevertheless, in our daily life, few of the facial expressions are exactly one of the predefined affective states. Therefore, to analyze the facial expressions more effectively, this paper proposes an Adaptive Group Lasso based Multilabel Regression approach, which depicts each facial expression with multiple continuous values of predefined affective states. Adaptive Group Lasso is adopted to depict the relationship between different labels which different facial expressions share some same affective facial areas (patches). Moreover, to solve the multi-label regression problem, a convex optimization formulation is presented, which would guarantee a global optimal solution. The experiment results based on JAFFE dataset have verified the superior performance of our approach.

Facial Expression Recognition (FER) techniques have already been adopted in numerous multimedia systems. Plenty of previous research assumes that each facial picture should be linked to only one of the predefined affective labels. Nevertheless, in practical applications, few of the expressions are exactly one of the predefined affective states. Therefore, to depict the facial expressions more accurately, this paper proposes a multi-label classification approach for FER and each facial expression would be labeled with one or multiple affective states. Meanwhile, by modeling the relationship between labels via Group Lasso regularization term, a maximum margin multi-label classifier is presented and the convex optimization formulation guarantees a global optimal solution. To evaluate the performance of our classifier, the JAFFE dataset is extended into a multi-label facial expression dataset by setting threshold to its continuous labels marked in the original dataset and the labeling results have shown that multiple labels can output a far more accurate description of facial expression. At the same time, the classification results have verified the superior performance of our algorithm.