Semantic representation of actions is essential for the development of mutual cognition towards efficient human-robot collaboration. As such, it has become an exciting emerging venue for robots to understand human intentions with Transformers, as a family of highly expressive models. However, due to high computational complexity, local redundancy in video frames can add up to inference latency of Transformers. To address this issue, we propose a prototype-aware token pruning method, namely ProtoPrune, to select important features for efficient action recognition. Specifically, for a video sequence, we first encode knowledge encapsulated in keyframes as prototype representations with a pretrained Transformer. Next, these prototypes teach the pretrained Transformer to preserve important visual features, pruning task-irrelevant tokens for improved throughput. In our experiments, the probabilistic token pruning method reduces over 37% of GFLOPs, with a performance retention rate of 92.9% without retraining ViT backbones. Our visualization showcases the improved robustness of learned action representations.

Leveraging large language models (LLMs) to fuse heterogeneous knowledge is an exciting emerging field. However, with billions of parameters, these pretrained language models are prohibitively computationally expensive at inference time. Token sparsification methods can proactively accelerate inference by selecting important features from the sequence but often require task-dependent retraining. To address this, we propose Bilevel Token prUniNg wiTh Infused kNowledGe (Bunting), an interpretable token pruning method that leverages task-level knowledge encoded in prefixes to guide token sparsification, eliminating the need for task-specific retraining. Bunting performs Bayesian Token Sparsification, where the inner loop learns a joint representation to perform the task, and the outer loop learns adaptive attention masks for sparse representations, thus pruning redundant tokens layer-by-layer without compromising the pretrained abilities of LLMs. Additionally, we introduce an innovative antiphrasis evaluation protocol to test model adaptivity on rhetorical relations. Furthermore, we demonstrate that precomputed prefixes can effectively guide token sparsification in different knowledge-intensive tasks, maintaining task-level knowledge to identify important tokens and reduce the finetuning burden. Experimental results demonstrate that our method achieves over $0.3 x$ wall-clock speed-up with only $0.14 \times$ learnable parameters in knowledge-intensive tasks. Our findings suggest that token pruning can improve out-of-distribution detection, with sarcasm being more challenging to detect than immorality.

Representation transfer is a widely used technique in natural language processing. We propose methods of cleaning the dominant dataset of text simplification (TS) WikiLarge in multi-views to remove errors that impact model training and fine-tuning. The results show that our method can effectively refine the dataset. We propose to take the pre-trained text representations from a similar task (e.g., text summarization) to text simplification to conduct a continue-fine-tuning strategy to improve the performance of pre-trained models on TS. This approach will speed up the training and make the model convergence easier. Besides, we also propose a new decoding strategy for simple text generation. It is able to generate simpler and more comprehensible text with controllable lexical simplicity. The experimental results show that our method can achieve good performance on many evaluation metrics.

Curriculum learning needs example difficulty to proceed from easy to hard. However, the credibility of image difficulty is rarely investigated, which can seriously affect the effectiveness of curricula. In this work, we propose Angular Gap, a measure of difficulty based on the difference in angular distance between feature embeddings and class-weight embeddings built by hyperspherical learning. To ascertain difficulty estimation, we introduce class-wise model calibration, as a post-training technique, to the learnt hyperbolic space. This bridges the gap between probabilistic model calibration and angular distance estimation of hyperspherical learning. We show the superiority of our calibrated Angular Gap over recent difficulty metrics on CIFAR10-H and ImageNetV2. We further propose a curriculum based on Angular Gap for unsupervised domain adaptation that can translate from learning easy samples to mining hard samples. We combine this curriculum with a state-of-the-art self-training method, Cycle Self Training (CST). The proposed Curricular CST learns robust representations and outperforms recent baselines on Office31 and VisDA 2017.