Long-horizon manipulation tasks with general instructions often implicitly encapsulate multiple sub-tasks, posing significant challenges in instruction following. While language planning is a common approach to decompose general instructions into stepwise sub-instructions, text-only guidance may lack expressiveness and lead to potential ambiguity. Considering that humans often imagine and visualize sub-instructions reasoning out before acting, the imagined subgoal images can provide more intuitive guidance and enhance the reliability of decomposition. Inspired by this, we propose **PERIA**(**PE**rceive, **R**eason, **I**magine, **A**ct), a novel framework that integrates holistic language planning and vision planning for long-horizon manipulation tasks with complex instructions, leveraging both logical and intuitive aspects of task decomposition. Specifically, we first perform a lightweight multimodal alignment on the encoding side to empower the MLLM to perceive visual details and language instructions. The MLLM is then jointly instruction-tuned with a pretrained image-editing model to unlock capabilities of simultaneous reasoning of language instructions and generation of imagined subgoals. Furthermore, we introduce a consistency alignment loss to encourage coherent subgoal images and align with their corresponding instructions, mitigating potential hallucinations and semantic conflicts between the two planning manners. Comprehensive evaluations across three task domains demonstrate that PERIA, benefiting from holistic language and vision planning, significantly outperforms competitive baselines in both instruction following accuracy and task success rate on complex manipulation tasks.

Past years, with the growth of smart-phones and applications, APP market has become an important mobile internet portal. As an important function in application market, APP search gains lots of attentions.However, mismatch between queries and APP is the most critical problem in APP search because of less text within term matching search engine. In this talk, we describe a semantic matching architecture in APP search--which mining topics and tags in big data. It enriches query and APP representations with topics and tags to achieve semantic matching in search. Some challenge must be considered: 1) How to extract tag-APP relationship from large web text. 2) How to use machine learning technologies to process de-noising and computing confidence. 3) How to hybrid ranking apps retrieved by different matching method. These will be introduced in some of our related works and as examples to describe how semantic matching is used in Tencent MyApp, an application market which serving hundreds of millions of users.

Hyperspectral image classification is a challenging problem. Among existing approaches to addressing this problem, the active learning (AL) and semisupervised learning (SSL) techniques have attracted much attention in recent years. AL usually involves a labor-intensive human-labeling process while SSL, although avoiding human labeling by assigning pseudolabels to unlabeled data, may introduce incorrect pseudolabels and thus deteriorate classification performance. To overcome these drawbacks, a novel approach named collaborative active and semisupervised learning (CASSL) is proposed in this paper. CASSL combines AL and SSL to invoke a collaborative labeling process by both human experts and classifiers. Specifically, an AL-based pseudolabel verification procedure is performed for gradually improving the pseudolabeling accuracy to facilitate SSL. Meanwhile, only those unlabeled data with low pseudolabeling confidence in SSL will become the query candidates in AL. We evaluate the performance of CASSL on three hyperspectral data sets and compare it with that of two state-of-the-art hyperspectral image classification methods. Experimental results reveal the superiority of CASSL.

Hyperspectral image classification is difficult due to the high dimensional features, high intraclass variance, low interclass variance but limited training samples. In this paper, the ECASSL (Ensured Collaborative Active and Semi-Supervised Labeling) approach, which attempts to exploit pseudo-labeled samples to improve the performance of active learning based hyperspectral image classification, is proposed. In detail, in each round of active query, we obtain new human labeled samples from active query strategy and pseudo-labeled samples from the current trained classifier collaboratively. After that, we update the classifier base on latest labeled and pseudo-labeled samples. And then we correct those pseudo-labels obtained from previous iterations with the new classifier. Finally, we train the final classifier base on both the labeled samples and pseudo-labeled samples. The experiment results show that our algorithm significantly reduced the need of labeled samples while achieving comparable performance when compared with state-of-the-art algorithms for hyperspectral image classification.