Multi-disease recognition is an important task in retinal fundus images, which enables efficient and low-cost diagnosis of ophthalmic disease. It is still challenging due to the difficulty and high expense of labeling medical images. To tackle this problem, we introduce a Text-Driven Multi-Modal Prototype Optimization (TMPO) approach, based on CLIP. By leveraging LLMs to generate hierarchical descriptions for multi-modal training pairs and establish fine-grained lesion correspondences across visual and textual representations, our TMPO approach enriches the semantic information of multi-disease labels and achieves more efficient embedding space alignment. This enables the generation of discriminative multi-modal prototypes within CLIP's embedding space. Extensive experiments on public and private datasets show that our approach effectively enhances diagnostic performance. Specifically, TMPO surpasses the SOTA methods, achieving the highest mAP of 0.7828 with a relative improvement of 5.04%. Our experiment code will be released later.

The Text-to-Video Retrieval (T2VR) task aims to retrieve unlabeled videos by textual queries with the same semantic meanings. Recent CLIP-based approaches have explored two frameworks: Two-Tower versus Single-Tower framework, yet the former suffers from low effectiveness, while the latter suffers from low efficiency. In this study, we explore a new Hybrid-Tower framework that can hybridize the advantages of the Two-Tower and Single-Tower framework, achieving high effectiveness and efficiency simultaneously. We propose a novel hybrid method, Fine-grained Pseudo-query Interaction and Generation for T2VR, ie, PIG, which includes a new pseudo-query generator designed to generate a pseudo-query for each video. This enables the video feature and the textual features of pseudo-query to interact in a fine-grained manner, similar to the Single-Tower approaches to hold high effectiveness, even before the real textual query is received. Simultaneously, our method introduces no additional storage or computational overhead compared to the Two-Tower framework during the inference stage, thus maintaining high efficiency. Extensive experiments on five commonly used text-video retrieval benchmarks demonstrate that our method achieves a significant improvement over the baseline, with an increase of $1.6\% \sim 3.9\%$ in R@1. Furthermore, our method matches the efficiency of Two-Tower models while achieving near state-of-the-art performance, highlighting the advantages of the Hybrid-Tower framework.

Nowadays, people usually turn to the Internet or auxiliary online-tools based on Visual Foundation Models when they recognize unfamiliar objects. The real-world image classification recognition is being greatly affected by learning paradigms based on Internet resources. In this paper, we propose a novel learning paradigm, named Retrieval Support Zero-Shot Learning (RSZSL), to reflect current human learning and recognition process in the Internet environment. RSZSL could cover many existing learning paradigms, in which a learner could use auxiliary Internet resources to optimize recognition and inference through real-time Internet searching. Furthermore, we propose an efficient Balanced Generative Framework as an advanced baseline method, which works with the retrieved Internet images, to complete RSZSL classification task. The framework can adapt to and perform well in different limited data scenarios, including the resource-rich but noisy and the resource-scarce scenario with zero-shot learning. Experiments on several traditional datasets and the retrieved auxiliary images demonstrate that our framework has superior robustness in dealing with real-world image recognition scenario and excellent performance in RSZSL classification task.

Recently, there has been a growing interest in applying self-supervised contrastive learning to medical image classification tasks. In this paper, we investigate its application for glaucoma classification using fundus images. Given the highly detailed features in fundus images, the effectiveness of contrastive learning, particularly when data augmentation is used to create contrastive pairs, proves to be notably limited. This constraint impairs the extraction of fine-grained visual representations on fundus images, resulting in reduced performance of self-supervised contrastive learning for glaucoma classification. To alleviate the above issues, we propose a pathology-aware representations contrastive learning (PaRCL) method, which learns more discriminative fine-grained representations compared to the previous self-supervised contrastive learning methods, to enhance the model performance for the glaucoma classification effectively. Furthermore, our method could focus more on the lesion area and effectively extract interpretable classification features. Particularly, to fully learn more discriminative visual representations, we introduce a pathology-aware subnetwork ensemble strategy (SES) as an alternative to generate contrastive pairs through traditional data augmentation techniques. Experiments on three clinical datasets demonstrate that our PaRCL could improve the performance of glaucoma classification on fundus images and could effectively focus on pathology areas of fundus images for disease diagnosis.

Recently, contrastive learning has received significant attention in various classification tasks of natural images. However, current contrastive learning frameworks display unsatisfactory performance on medical images due to the inability of obtaining fine-grained visual features. In this paper, we propose a Lesion-Aware Contrastive Learning (LACL) framework to learn more discriminative and comprehensive representations and reinforce the attention of diagnosis regions on medical images. LACL framework includes two phases of training procedures: the comprehensive feature-extracting phase and the contrastive learning enhancement phase. In the first phase, LACL fully captures meaningful deep features related to the training targets to form comprehensive visual representations, by training a novel lesion-aware module we proposed. In the second phase, we introduce the previous representation information into contrastive learning to guide the LACL framework in learning disease- related features. This approach provides more effective guidance than the traditional contrastive learning method of directly comparing features. Extensive experiments on several benchmark datasets demonstrate that our LACL framework significantly improves the performance of medical image classification and highlights the lesion areas for disease diagnosis.

The retinal nerve fiber layer defects (RNFLD) provide early objective evidence for many retinal abnormalities, especially early glaucoma. The recent success of deep learning has led to exciting prospects in automating the detection of RNFLD, but it is highly dependent on the availability of large-scale datasets carefully annotated by experienced ophthalmologists, which is both time-consuming and cost-prohibitive. Most previous works on RNFLD detection lack a unified annotation scheme. In addition, they fail to exploit the intrinsic morphological characteristics of retinal nerve fiber bundles (RNFB). This work presents an automatic RNFB tracing method that is applicable not only in automating the RNFLD annotation process, but also in performing a quasi-polar transformation on fundus images for subsequent RNFLD detection tasks. Also, our proposed method paves the way for alleviating the problem of inter-annotator variation incurred merely by different annotation habits among annotators and reducing the noise before the input stage. Experiments reveal that our proposed quasi-polar transformation provides a significant boost to the existing RNFLD detection method and surpasses the state-of-the-art F1 by 4.2%, and that our method also offers a more accurate and reasonable description of RNFLD detection results, which demonstrates the orientation of RNFLD rather than vague position indication.

Optic disc hemorrhage (ODH) is an important lesion factor for eye disease diagnosis. ODH has some specific displaying characteristics on fundus images, including covering fuzzy small domains and displaying similar to vessels, which greatly increase ODH detection difficulty. In this paper, we propose a novel position-guided attention network to detect optic disc hemorrhage on fundus images. Our method greatly takes advantage of the prior knowledge of ODH position information and the multitask learning dependencies related to the ODH segmentation and ODH classification, to build a position information attention module and a correlative feature fusion module. Moreover, we introduce a disc edge strength map on the optic disc to constrain the attention domains. Due to the limitation of ODH labeled data, an online hard case segmentation strategy on small samples are proposed to train our method. Experiments show that our method could greatly reduce the detection of false positives when detecting ODH on fundus images, so as to obtain superior performance on the ODH detection with small samples. Further, a fundus image dataset with professional ODH labels is published to advance the research of ODH detection (https://github.com/JieGenius/disc_hemo_dataset).

The optic disc and cup contour detection task has been studied extensively, as it is the predominant attribute to calculate cup-to-disc ratio (CDR) and further diagnose glaucoma. Currently, methods for optic disc and cup contour detection task mostly include two types: the coordinate point regression and the probability map classification. Normally, the latter is better. However, the probability map is unfavorable to be directly applied to the optic disc and cup contour detection task, which makes the model outputs more than one optic disc and cup in some cases, and uneven results. In addition, the probability map classification methods can not directly optimize the calculation of CDR in the training process, which further limits the performance of neural networks. In this paper, we design a new optic disc and cup contour representation method called Contour Offset Map to solve the two mentioned problems. Our approach can be applied seamlessly to the existing semantic segmentation networks by only modifying the loss functions and the number of output maps rather than the network structure. Moreover, a rim loss and a CDR loss are introduced to further improve the performance of our method in the training process. Experiments on our private dataset and a public dataset demonstrate the superior performance of our Contour Offset Map on the task of the optic disc and cup contour detection.

With the purpose of addressing the scarcity of attribute diversity in Zero-shot Learning (ZSL), we propose to search for additional attributes in embedding space to extend the class embedding, providing a more discriminative representation of the class prototype. Meanwhile, to tackle the inherent noise behind manually annotated attributes, we apply multi-layer convolutional processing on semantic features rather than conventional linear transformation for filtering. Moreover, we employ Center Loss to assist the training stage, which helps the learned mapping be more accurate and consistent with the corresponding class's prototype. Combining these modules mentioned above, extensive experiments on several public datasets show that our method could yield decent improvements. This proposed way of extending attributes can also be migrated to other models or tasks and obtain better results.

This paper proposes a real-time deepfake framework to assist users use deep forgery to conduct live streaming, further to protect privacy and increase interesting by selecting different reference faces to create a non-existent fake face. Nowadays, because of the demand for live broadcast functions such as selling goods, playing games, and auctions, the opportunities for anchor exposure are increasing, which leads live streamers pay more attention to their privacy protection. Meanwhile, the traditional technology of deepfake is more likely to infring on the portrait rights of others, so our framework supports users to select different face features for facial tampering to avoid infringement. In our framework, through feature extractor, heatmap transformer, heatmap regression and face blending, face reenactment could be confirmed effectively. Users can enrich the personal face feature database by uploading different photos, and then select the desired picture for tampering on this basis, and finally real-time tampering live broadcast is achieved. Moreover, our framework is a closed loop self-adaptation system as it allows users to update the database themselves to extend face feature data and improve conversion efficiency.

Automatic artery/vein (Arkers for the early diagnosis of many systemic diseases. Unfortunately, current methods have some limitations in AN segmentation, especially the lack of annotated data and the serious data imbalance. Thus, A novel multimodal multiscale fusion network (MMF-Net) is proposed to alleviate the above problems, which utilizes the internal semantic information of vessels adequately to enhance the AN segmentation. Particularly, the MMF-Net introduces a multimodal (MM) module that could highlight the vessel structure from the original fundus image to constrain the AN image features, which reduces the influence of background noise. In addition, the MMF-Net exploits a multiscale transformation (MT) module to extract the vessel information efficiently from the multimodal feature representations. Finally, A multi-feature fusion (MF) module is applied in MMF-Net to split and reorganize the pixel feature from different scales to improve the robustness of AN segmentation. Experiments on two public benchmark datasets show that our method has achieved superior performance and surpassed other existing state-of-the-art methods in the accuracy of AN segmentation.

This paper attacks the challenging problem of video retrieval by text. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described exclusively in the form of a natural-language sentence, with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is crucial. To that end, the two modalities need to be first encoded into real-valued vectors and then projected into a common space. In this paper we achieve this by proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Our novelty is two-fold. First, different from prior art that resorts to a specific single-level encoder, the proposed network performs multi-level encoding that represents the rich content of both modalities in a coarse-to-fine fashion. Second, different from a conventional common space learning algorithm which is either concept based or latent space based, we introduce hybrid space learning which combines the high performance of the latent space and the good interpretability of the concept space. Dual encoding is conceptually simple, practically effective and end-to-end trained with hybrid space learning. Extensive experiments on four challenging video datasets show the viability of the new method. Code and data are available at https://github.com/danieljf24/hybrid_space.

Recently, semi-supervision has been successfully applied to convolutional neural networks, significantly improving the results of many computer vision tasks. Unfortunately, research of glaucoma diagnosis rarely attempts to introduce the idea of semi-supervision. In particular, various datasets used for glaucoma classification have the problem of domain inconsistency. Therefore, it is difficult to train a widely applicable model that can achieve good performance in multiple datasets. To solve this problem, a multi-level amplified iterative training method for glaucoma diagnosis is proposed based on the potential of semi-supervision to boost the robustness of the model. Specifically, our training method expands the content of self-training and knowledge distillation, so that the model can overcome the interference of pseudo labels while using unlabeled data, which makes the model focus on learning the features that are helpful to the task and continuously increase its understanding of glaucoma images and non-glaucoma images. Experiments show that our multi-level amplified iterative training can significantly improve the accuracy and robustness of glaucoma diagnosis.

Predicting the relevance between two given videos with respect to their visual content is a key component for content-based video recommendation and retrieval. Thanks to the increasing availability of pre-trained image and video convolutional neural network models, deep visual features are widely used for video content representation. However, as how two videos are relevant is task-dependent, such off-the-shelf features are not always optimal for all tasks. Moreover, due to varied concerns including copyright, privacy and security, one might have access to only pre-computed video features rather than original videos. We propose in this paper feature re-learning for improving video relevance prediction, with no need of revisiting the original video content. In particular, re-learning is realized by projecting a given deep feature into a new space by an affine transformation. We optimize the re-learning process by a novel negative-enhanced triplet ranking loss. In order to generate more training data, we propose a new data augmentation strategy which works directly on frame-level and video-level features. Extensive experiments in the context of the Hulu Content-based Video Relevance Prediction Challenge 2018 justify the effectiveness of the proposed method and its state-of-the-art performance for content-based video relevance prediction.

Retrieving unlabeled videos by textual queries, known as Ad-hoc Video Search (AVS), is a core theme in multimedia data management and retrieval. The success of AVS counts on cross-modal representation learning that encodes both query sentences and videos into common spaces for semantic similarity computation. Inspired by the initial success of previously few works in combining multiple sentence encoders, this paper takes a step forward by developing a new and general method for effectively exploiting diverse sentence encoders. The novelty of the proposed method, which we termSentence Encoder Assembly(SEA), is two-fold. First, different from prior art that uses only a single common space, SEA supports text-video matching in multiple encoder-specific common spaces. Such a property prevents the matching from being dominated by a specific encoder that produces an encoding vector much longer than other encoders. Second, in order to explore complementarities among the individual common spaces, we propose multi-space multi-loss learning. As extensive experiments on four benchmarks (MSR-VTT, TRECVID AVS 2016-2019, TGIF and MSVD) show, SEA surpasses the state-of-the-art. In addition, SEA is extremely ease to implement. All this makes SEA an appealing solution for AVS and promising for continuously advancing the task by harvesting new sentence encoders.

Expanding visual categorization into a novel domain without the need of extra annotation has been a long-term interest for multimedia intelligence. Previously, this challenge has been approached by unsupervised domain adaptation (UDA). Given labeled data from a source domain and unlabeled data from a target domain, UDA seeks for a deep representation that is both discriminative and domain-invariant. While UDA focuses on the target domain, we argue that the performance on both source and target domains matters, as in practice which domain a test example comes from is unknown. In this article, we extend UDA by proposing a new task called unsupervised domain expansion (UDE), which aims to adapt a deep model for the target domain with its unlabeled data, meanwhile maintaining the model’s performance on the source domain. We propose Knowledge Distillation Domain Expansion (KDDE) as a general method for the UDE task. Its domain-adaptation module can be instantiated with any existing model. We develop a knowledge distillation-based learning mechanism, enabling KDDE to optimize a single objective wherein the source and target domains are equally treated. Extensive experiments on two major benchmarks, i.e., Office-Home and DomainNet, show that KDDE compares favorably against four competitive baselines, i.e., DDC, DANN, DAAN, and CDAN, for both UDA and UDE tasks. Our study also reveals that the current UDA models improve their performance on the target domain at the cost of noticeable performance loss on the source domain.

Domain adaptation is an important research topic in the field of computer vision, where the goal is to solve the difference of data distribution between different scenarios of the same task. In recent times, adversarial learning method becomes a mainstream approach to generate complicated images across diverse domains through optimizing deep networks, and it can also improve the recognition accuracy rate of deep networks despite existing domain shift or dataset bias. However, there are few effective efforts of domain adaptation for the disease diagnosis on fundus images. Fundus images are normally captured on different medical devices with different rules. When diagnosing glaucoma, there is a serious homogeneous domain shift, which means feature spaces between target domain and source domain images have a distribution shift although they are very similar. We propose a unified framework to solve this problem. Previous studies have shown that glaucoma can be monitored by analyzing the optic disc/cup and its surroundings. So we exploit a novel reconstruction loss which not only leverages unsupervised data to bring the source and target distributions closer but also keeps original target domain images label unchanged. The experimental results on several public and private datasets demonstrate that our method could increase the classification accuracy of glaucoma diagnosis.

On account of a large scale of dataset need to be annotated to fit for specific tasks, Zero-Shot Learning(ZSL) has invoked so much attention and got significant progress in recent research due to the prevalence of deep neural networks. At present, ZSL is mainly solved through the utilization of auxiliary information, such as semantic attributes and text descriptions. And then, we can employ the mapping method to bridge the gap between visual and semantic space. However, due to the lack of effective use of auxiliary information, this problem has not been solved well. Inspired by previous work, we consider that visual space can be used as the embedding space to get a stronger ability to express the precise characteristics of semantic information. Meanwhile, we take into account that there are some noise attributes in the annotated information of public datasets that need to be processed. Based on these considerations, we propose an end-to-end method with convolutional architecture, instead of conventionally linear projection, to provide a deep representation for semantic information to solve ZSL. Semantic features would express more detailed and precise information after being feed into our method. Besides, we use word embedding to generate some superclasses for original classes and propose a new loss function for these superclasses to assist in training. Experiments show that our method can get decent improvements for ZSL and Generalized Zero-Shot Learning(GZSL) on several public datasets.

This paper attacks the challenging problem of zero-example video retrieval. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described in natural language text with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is required. The majority of existing methods are concept based, extracting relevant concepts from queries and videos and accordingly establishing associations between the two modalities. In contrast, this paper takes a concept-free approach, proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Dual encoding is conceptually simple, practically effective and end-to-end. As experiments on three benchmarks, i.e. MSR-VTT, TRECVID 2016 and 2017 Ad-hoc Video Search show, the proposed solution establishes a new state-of-the-art for zero-example video retrieval.

Ad-hoc video search (AVS) is an important yet challenging problem in multimedia retrieval. Different from previous concept-based methods, we propose a fully deep learning method for query representation learning. The proposed method requires no explicit concept modeling, matching and selection. The backbone of our method is the proposed W2VV++ model, a super version of Word2VisualVec (W2VV) previously developed for visual-to-text matching. W2VV++ is obtained by tweaking W2VV with a better sentence encoding strategy and an improved triplet ranking loss. With these simple yet important changes, W2VV++ brings in a substantial improvement. As our participation in the TRECVID 2018 AVS task and retrospective experiments on the TRECVID 2016 and 2017 data show, our best single model, with an overall inferred average precision (infAP) of 0.157, outperforms the state-of-the-art. The performance can be further boosted by model ensemble using late average fusion, reaching a higher infAP of 0.163. With W2VV++, we establish a new baseline for ad-hoc video search.

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20 342 images annotated with 27 218 Chinese sentences and 70 993 tags, COCO-CN is currently the largest Chinese-English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning, and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn.

This paper describes our solution for the Hulu Content-based Video Relevance Prediction Challenge. Noting the deficiency of the original features, we propose feature re-learning to improve video relevance prediction. To generate more training instances for supervised learning, we develop two data augmentation strategies, one for frame-level features and the other for video-level features. In addition, late fusion of multiple models is employed to further boost the performance. Evaluation conducted by the organizers shows that our best run outperforms the Hulu baseline, obtaining relative improvements of 26.2% and 30.2% on the TV-shows track and the Movies track, respectively, in terms of [email protected] The results clearly justify the effectiveness of the proposed solution.

Generalized zero-shot learning (GZSL) aims to recognize any test instance coming either from a known class or from a novel class that has no training instance. To synthesize training instances for novel classes and thus resolving GZSL as a common classification problem, we propose a Dissimilarity Representation Learning (DSS) method. Dissimilarity representation is to represent a specific instance in terms of its (dis)similarity to other instances in a visual or attribute based feature space. In the dissimilarity space, instances of the novel classes are synthesized by an end-to-end optimized neural network. The neural network realizes two-level feature mappings and domain adaptions in the dissimilarity space and the attribute based feature space. Experimental results on five benchmark datasets, i.e., AWA, AWA$_2$, SUN, CUB, and aPY, show that the proposed method improves the state-of-the-art with a large margin, approximately 10% gain in terms of the harmonic mean of the top-1 accuracy. Consequently, this paper establishes a new baseline for GZSL.

Zero-shot learning (ZSL) which aims to recognize unseen classes with no labeled training sample, efficiently tackles the problem of missing labeled data in image retrieval. Nowadays there are mainly two types of popular methods for ZSL to recognize images of unseen classes: probabilistic reasoning and feature projection. Different from these existing types of methods, we propose a new method: sample construction to deal with the problem of ZSL. Our proposed method, called Imagination Based Sample Construction (IBSC), innovatively constructs image samples of target classes in feature space by mimicking human associative cognition process. Based on an association between attribute and feature, target samples are constructed from different parts of various samples. Furthermore, dissimilarity representation is employed to select high-quality constructed samples which are used as labeled data to train a specific classifier for those unseen classes. In this way, zero-shot learning is turned into a supervised learning problem. As far as we know, it is the first work to construct samples for ZSL thus, our work is viewed as a baseline for future sample construction methods. Experiments on four benchmark datasets show the superiority of our proposed method.

In order to take advantages of evolutionary algorithms inspired by different biological evolutions, varieties of approaches have been proposed to combine them together. One of them is the portfolio approach, which keeps choosing a component algorithm from a portfolio of evolutionary algorithms (EAs) to run during the optimizing process. In our approach, each component algorithm has its own population and runs independently without information exchange. At the beginning of each generation, only the component algorithm with the best predicted performance is allowed to run. The proposed portfolio approach is tested on the CEC2016 real-parameter single objective optimization benchmarks. The results show that it is competitive.

Various classifiers based on the machine learning techniques have been widely used in security applications. Meanwhile, they also became an attack target of adversaries. Many existing studies have paid much attention to the evasion attacks on the online classifiers and discussed defensive methods. However, the security of the classifiers deployed in the client environment has not got the attention it deserves. Besides, earlier studies only concentrated on the experimental classifiers developed for research purposes only. The security of widely-used commercial classifiers still remains unclear. In this paper, we use the Google's phishing pages filter (GPPF), a classifier deployed in the Chrome browser which owns over one billion users, as a case to investigate the security challenges for the client-side classifiers. We present a new attack methodology targeting on client-side classifiers, called classifiers cracking. With the methodology, we successfully cracked the classification model of GPPF and extracted sufficient knowledge can be exploited for evasion attacks, including the classification algorithm, scoring rules and features, etc. Most importantly, we completely reverse engineered 84.8% scoring rules, covering most of high-weighted rules. Based on the cracked information, we performed two kinds of evasion attacks to GPPF, using 100 real phishing pages for the evaluation purpose. The experiments show that all the phishing pages (100%) can be easily manipulated to bypass the detection of GPPF. Our study demonstrates that the existing client-side classifiers are very vulnerable to classifiers cracking attacks.

With the increasing use of audio sensors in user generated content collection, how to detect semantic concepts using audio streams has become an important research problem. In this paper, we present a semantic concept annotation system using soundtracks/ audio of the video. We investigate three different acoustic feature representations for audio semantic concept annotation and explore fusion of audio annotation with visual annotation systems. We test our system on the data collection from HUAWEI Accurate and Fast Mobile Video Annotation Grand Challenge 2014. The experimental results show that our audio-only concept annotation system can detect semantic concepts significantly better than random guess. It can also provide significant complementary information to the visual-based concept annotation system for performance boost. Further detailed analysis shows that for interpreting a semantic concept both visually and acoustically, it is better to train concept models for the visual system and audio system using visual-driven and audio-driven ground truth separately.

With the increasing use of audio sensors in user generated content (UGC) collections, semantic concept annotation from video soundtracks has become an important research problem. In this paper, we investigate reducing the semantic gap of the traditional data-driven bag-of-audio-words based audio annotation approach by utilizing the large-amount of wild audio data and their rich user tags, from which we propose a new feature representation based on semantic class model distance. We conduct experiments on the data collection from HUAWEI Accurate and Fast Mobile Video Annotation Grand Challenge 2014. We also fuse the audio-only annotation system with a visual-only system. The experimental results show that our audio-only concept annotation system can detect semantic concepts significantly better than does random guessing. The new feature representation achieves comparable annotation performance with the bag-of-audio-words feature. In addition, it can provide more semantic interpretation in the output. The experimental results also prove that the audio-only system can provide significant complementary information to the visual-only concept annotation system for performance boost and for better interpretation of semantic concepts both visually and acoustically.

This paper proposed a framework to assist highlighting and annotating music utilization situation automatically within videos, further to supervise and protect music copyrights. Nowadays, music copyrighters pay attention to their rights increasingly, thus music embedded in TV channel videos should be validated to avoid infringing. Our framework supports Music Copyrighter Society of China(MCSC) to do statistic works to protect the copyright owners. In our framework, through AV separation, feature extractor, classification and assemblage functions, music positioning could be confirmed effectively. Then applying music fingerprint retrieving, music could be annotated automatically with high accuracy. Moreover, our framework is a closed loop self-adaptation system as it can be re-trained regularly to expand annotation database and enhance classifier's efficiency. The system based on our framework has been implemented in MCSC and its effectiveness has been evaluated in a real-life scenario. The results, on experiments of the real-life TV stations and comparisons of former works, show that the music positioning and annotation completed automatically by our system have significant improvement about over 30 times enhancement on the working efficiency.

Given the difficulty of acquiring labeled examples for many fine-grained visual classes, there is an increasing interest in zero-shot image tagging, aiming to tag images with novel labels that have no training examples present. Using a semantic space trained by a neural language model, the current state-of-the-art embeds both images and labels into the space, wherein cross-media similarity is computed. However, for labels of relatively low occurrence, its similarity to images and other labels can be unreliable. This paper proposes Hierarchical Semantic Embedding (HierSE), a simple model that exploits the WordNet hierarchy to improve label embedding and consequently image embedding. Moreover, we identify two good tricks, namely training the neural language model using Flickr tags instead of web documents, and using partial match instead of full match for vectorizing a WordNet node. All this lets us outperform the state-of-the-art. On a test set of over 1,500 visual object classes and 1.3 million images, the proposed model beats the current best results (18.3% versus 9.4% in hit@1).

In this paper, we propose an artificial bee group colony algorithm for numerical function optimization, based on the thoughts of group competition and similar property characteristic. Our algorithm contains three optimization strategies including grouping strategy, similar property strategy and competition strategy, which could not only ensure the algorithm finds better solutions stably, but also induce the algorithm to maintain solution diversification. Moreover, the similar property strategy could produce efficient exploring to find better solutions with skipping optimization. We evaluated the performance of our proposed algorithm on some standard numerical benchmark functions. The results demonstrate that our algorithm is able to yield higher quality solutions with faster convergence than either the original ABC or some other authoritative swarm intelligent algorithms.

We consider the problem of simplifying the artificial bee colony (ABC) algorithm and study the mechanism that actually contributes to its outstanding performance. Based on the collective behaviors of honey bee colony, ABC algorithm is one of the most effective algorithms solving numeric optimization problem. It simulates the collective behavior of honey bee colony with a set of behavioral rules observed by individuals of the swarm. Most of the previous researches on this topic focus on adding features to the algorithm in order to improve its performance on particular problems. In contrast, the work presented in this paper focus on removing unnecessary behavioral rules from the basic ABC algorithm, and investigate how simple the behavior rules can be without qualitative change of its performance on frequently-used benchmarks.

Despite the impressive progress in music emotion recognition, it remains unclear what aspect of a song, i.e., singing voice and accompanied music, carries more emotional information. As an initial attempt to answer the question, we introduce source separation into a standard music emotion recognition system. This allows us to compare systems with and without source separation, and consequently reveal the influence of singing voice and accompanied music on emotion recognition. Classification experiments on a set of 267 songs with last.fm annotations verify the new finding that source separation improves song music emotion recognition.

In this paper, a novel hybrid evolutionary algorithm combining a Hopfield net and a local search strategy is proposed to solve maximum clique problem. The algorithm makes full use of powerful searching capability of Hopfield net and probabilistic statistic feature of estimation of distribution algorithm to produce wider search in global solution domain. In particular, a possible extension way correlated with local search optimization is introduced to affect the mutation probability thus to produce guided evolution. Experiments on the popular DIMACS benchmark demonstrate that the hybrid evolutionary algorithm produces comparable and better results than other compared algorithms, including EA/G which is a state-of-the-art algorithm in the field of evolutionary computation.

Recently, social tagging systems become more and more popular in many Web 2.0 applications. In such systems, Users are allowed to annotate a particular resource with a freely chosen a set of tags. These user-generated tags can represent users' interests more concise and closer to human understanding. Interests will change over time. Thus, how to describe users' interests and interests transfer path become a big challenge for personalized recommendation systems. In this approach, we propose a variable-length time interval division algorithm and user interest model based on time interval. Then, in order to draw users' interests transfer path over a specific time period, we suggest interest transfer model. After that, we apply a classical community partition algorithm in our approach to separate users into communities. Finally, we raise a novel method to measure users' similarities based on interest transfer model and provide personalized tag recommendation according to similar users' interests in their next time intervals. Experimental results demonstrate the higher precision and recall with our approach than classical user-based collaborative filtering methods.

Elastic net is an efficient neural network algorithm to solve combinational optimization problems, especially to solve traveling salesman problem. However, aimed to solve large problems, elastic net illustrates insufficient solving capability. Based on our observations and analysis of router properties in the optimal/near-optimal solutions of TSP, we introduce a novel neural network algorithm named unsupervised up-to-bottom hierarchical clustering elastic net (UBHCE) to solve TSP in parallel. Combined with the remarkable geometrical property of elastic net, the UBHCE partitions TSP hierarchically through utilizing an embedded suitable clustering method (UBHC), which is able to decrease problem complexity gradually. Through summarizing and analyzing the coefficient setting regularity of activity function in elastic net, we present a flexible coefficient tuning strategy to adapt to the UBHCE for the process of gradually decreasing TSP. The experimental results on a large amount of instances of random TSP and benchmark TSP suggest that the UBHCE has a higher average success rate for obtaining globally optimal/near-optimal solutions, moreover, it is more suitable to deal with complex problems in parallel.

Search engine is a common tool to retrieve the information in the Web. But the current status of returned results is still far from satisfaction. Users have to be confronted with searching for a long result list to get the information really wanted. Many works focused on the post processing search results to facilitate users to examine the results. One of the common ways of post processing search result is clustering. Term-based clustering appears as first way to cluster the results. But this method is suffering from the poor quality while the processed pages have little text. Link-based clustering can conquer this problem. But the quality of clusters heavily depends on the number of in-links and out-links in common. In this paper, we propose that the short text attached to in-link is valuable information and it is helpful to reach high clustering quality. To distinguish them with general snippet, we name it as in-snippet. Based on the in-snippet, we propose a new clustering method that combines the links and the in-snippets together. In our method, similarity between pages consists of two parts : link similarity and term similarity. We designed related algorithm to implement clustering. In order to prevent bias from human judgments, the experiment datasets are collected from Open Directory Project(DMOZ). Due to DMOZ is human-edited directory, the datasets from DMOZ has higher quality and larger scale. We use entropy and f-measure to evaluate the quality of the final clusters. By being compared with the link-based and the pure term-based algorithms, our method outperforms others in clustering quality.

In this paper, we analyze the relationships between solution qualities and the number of initial dynamic points in elastic net algorithm, and propose a rebuilt clone elastic network algorithm(ReBCEN) for traveling salesman problem(TSP) according to the analysis results. For solving traveling salesman problem, firstly our algorithm initializes less dynamic points to produce a general accessing path of solution which is similar with that of optimal solutions. Then by using our novel parameter tuning strategies and nearest neighbor method, our algorithm performs a rebuilt cloning process to insert cloned dynamic points into the rubber band of elastic net and adjust accessing path to find the optimal solutions. As it utilizes less dynamic points in every epoch and unmatched cities in TSP are reduced gradually, ReBCEN can obtain faster convergence speed than the original elastic net and some other modified elastic nets. ReBCEN was simulated on some random TSPs and TSPLIB instances to verify its performance by comparison with other algorithms.

Based on the analysis and comparison of several annealing strategies, we present a flexible annealing chaotic neural network which has flexible controlling ability and quick convergence rate to optimization problem. The proposed network has rich and adjustable chaotic dynamics at the beginning, and then can converge quickly to stable states. We test the network on the maximum clique problem by some graphs of the DIMACS clique instances, p-random and k random graphs. The simulations show that the flexible annealing chaotic neural network can get satisfactory solutions at very little time and few steps. The comparison between our proposed network and other chaotic neural networks denotes that the proposed network has superior executive efficiency and better ability to get optimal or near-optimal solution.