Predicting labels directly from data has been the standard in label learning tasks, e.g., supervised learning, where models often prioritize feature compression and extraction from inputs under the assumption that label information is less complex. However, recent prediction tasks often face predicting complex labels, exacerbating the challenge of learning mappings from learned features to high-fidelity label representations. To this end, we draw inspiration from the consistency training concept in generative consistency models and propose predictive consistency learning (PCL), a novel learning paradigm that decomposes the full label information into a progressive learning procedure, mitigating the label capture challenge. Besides data inputs, PCL additionally receives input from noise-perturbed labels as an additional reference, pursuing predictive consistency across different noise levels. It simultaneously learns the relationship between latent features and a spectrum of label information, which enables progressive learning for complex predictions and allows multi-step inference analogous to gradual denoising, thereby enhancing the prediction quality. Experiments on vision, text, and graph tasks show the superiority of PCL over conventional supervised training in complex label prediction tasks.

We rethink the basic operations, i.e., inner product and matrix multiplication used in neural networks. A quantum-inspired alternative is proposed, utilizing the power of high-dimensional Hilbert space by devising a high-order form of tensor product. We re-parameterize the original (low-order) vectors/matrices into an expressive high-order form, without incurring extra model parameters, and the extra computational overhead is negligible (e.g., about 2%). As an in-place transparent atomic operation, we show its use in the key components in Transformers: token embeddings, attentions (query, key, value) and the MLP. Due to its inherent compatibility to vanilla multiplicative operations, we propose C2Q-SFT, i.e., classic-to-quantum (C2Q) protocol for supervised fine-tuning (SFT): it continues to train a given model by transparently replacing the standard operations with ours. As shown by our experiments, it shows advantages for both training from scratch and fine-tuning on downstream tasks across scales of LLMs. C2Q-SFT consistently outperforms standard SFT, with relative improvements on MMLU (+0.56%) and GSM8k (+0.61%). It sheds light on the innovation of operations in networks, orthogonal to the efforts on new architecture, position encoding, and training algorithms, etc. See project page at: https://github.com/Thinklab-SJTU/LLM/QI-LLM.