Hyperparameters are critical to LLM reinforcement learning (RL), but existing hyperparameter optimization (HPO) methods remain inefficient in this area, due to the massive model scale and resource-intensive training cycles.In this paper, we propose Joint Fidelity Hyperparameter Optimization (JF-HPO), which simultaneously adapts both model size and training budget as fidelity.JF-HPO is empowered by: (i) a small proxy model of the target LLM for efficient training and evaluation in each HPO trial; (ii) several carefully designed early-stopping strategies based on training dynamics; (iii) an efficient checkpointing mechanism to eliminate redundant computations.JF-HPO significantly improves the computational efficiency of each trial (up to 14.9×) compared with existing HPO methods, thus achieving better predictive accuracy in most cases under the same time budget.Notably, JF-HPO delivers performance improvements ranging from 5.8% to 111.6% over VeRL Recipe.

As the size of language models notably grows, fine-tuning the models becomes more challenging: fine-tuning with first-order optimizers (e.g., SGD and Adam) requires high memory consumption, while fine-tuning with a memory-efficient zeroth-order optimizer (MeZO) has a significant accuracy drop and slower convergence rate. In this work, we propose a Low order Hybrid Optimizer (LoHO) which merges zeroth-order (ZO) and first-order (FO) optimizers for fine-tuning. LoHO is empowered with inter-layer hybrid optimization and intra-layer hybrid optimization, which boosts the accuracy of MeZO while keeping memory usage within a budget. The inter-layer hybrid optimization exploits the FO optimizer in deep layers and the ZO optimizer in shallow ones, therefore avoiding unnecessary gradient propagation to improve memory efficiency. The intra-layer hybrid optimization updates a proportion of parameters in a layer by the ZO optimizer, and the rest by the FO optimizer, taking advantage of gradient sparsity for high efficiency implementation. Our experimental results across common datasets on different pre-trained backbones (i.e., RoBERTa-large, OPT-13B and OPT-30B) demonstrate that LoHO can significantly improve the predictive accuracy and convergence rate of MeZO, while controlling the memory footprint during fine-tuning. Moreover, LoHO can achieve comparable performance with first-order fine-tuning using substantially fewer memory resources.

Parameter-efficient fine-tuning (PEFT) of pre-trained language models (PLMs) has shown promising results by updating significantly fewer parameters than full fine-tuning. Masking-based fine-tuning is one type of PEFT methods, by freezing the majority of the model parameters during fine-tuning. Existing masking-based fine-tuning methods either need to manually select the trainable parameters (heuristic-based), or perform mask learning to adaptively select the trainable parameters with high memory and computation cost. To address these problems, this paper proposes Low-Rank based Efficient Mask Learning (LoReML). LoReML performs mask learning based on low-rank decomposition and matrix reconstruction with a small ratio of new parameters. After mask learning, LoReML uses the scaled intermediate results in mask learning as warm start initialization to boost the model quality, then freezes the masked parameters accordingly, and fine-tunes the PLM. Moreover, LoReML exploits sparse training techniques to enhance the memory efficiency in masking-based fine-tuning. Experimental results across various tasks and pre-trained backbones demonstrate that LoReML can notably outperform existing heuristic-based methods. Moreover, LoReML achieves competitive or better performance compared with the adaptive mask learning methods, while improving memory and computation efficiency by over 50% in mask learning.

Multimodal sentiment analysis aims to predict sentiment of language text with the help of other modalities, such as vision and acoustic features.Previous studies focused on learning the joint representation of multiple modalities, ignoring some useful knowledge contained in language modal.In this paper, we try to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features.Our method consists of two components: shallow fusion part and aggregation part.For the shallow fusion part, we use crossmodal coattention mechanism to obtain bidirectional context information of each two modals to get the fused shallow representations.For the aggregation part, we design a multitask of sentimental words classification to help and guide the deep fusion of the three modalities and obtain the final sentimental words aware fusion representation.We carry out several experiments on CMU-MOSI, CMU-MOSEI and YouTube datasets.The experimental results show that introducing sentimental words prediction as a multitask can really improve the fusion representation of multiple modalities.