Hongyuan Dong, Zijian Kang, Weijie Yin, LiangXiao LiangXiao, ChaoFeng ChaoFeng, Ran Jiao. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Multi-modal keyphrase prediction (MMKP) aims to advance beyond text-only methods by incorporating multiple modalities of input information to produce a set of conclusive phrases.Traditional multi-modal approaches have been proven to have significant limitations in handling the challenging absence and unseen scenarios.Additionally, we identify shortcomings in existing benchmarks that overestimate model capability due to significant overlap in training tests.In this work, we propose leveraging vision-language models (VLMs) for the MMKP task.Firstly, we use two widely-used strategies, e.g., zero-shot and supervised fine-tuning (SFT) to assess the lower bound performance of VLMs.Next, to improve the complex reasoning capabilities of VLMs, we adopt Fine-tune-CoT, which leverages high-quality CoT reasoning data generated by a teacher model to finetune smaller models.Finally, to address the "overthinking" phenomenon, we propose a dynamic CoT strategy which adaptively injects CoT data during training, allowing the model to flexibly leverage its reasoning capabilities during the inference stage.We evaluate the proposed strategies on various datasets and the experimental results demonstrate the effectiveness of the proposed approaches.The code is available at https://github.com/bytedance/DynamicCoT. 6.

Learning rate is widely regarded as crucial for effective foundation model pretraining. Recent research explores and demonstrates the transferability of learning rate configurations across varying model and dataset sizes, etc. Nevertheless, these approaches are constrained to specific training scenarios and typically necessitate extensive hyperparameter tuning on proxy models. In this work, we propose \textbf{AdaLRS}, a plug-in-and-play adaptive learning rate search algorithm that conducts online optimal learning rate search via optimizing loss descent velocities. We provide theoretical and experimental analyzes to show that foundation model pretraining loss and its descent velocity are both convex and share the same optimal learning rate. Relying solely on training loss dynamics, AdaLRS involves few extra computations to guide the search process, and its convergence is guaranteed via theoretical analysis. Experiments on both LLM and VLM pretraining show that AdaLRS adjusts suboptimal learning rates to the neighborhood of optimum with marked efficiency and effectiveness, with model performance improved accordingly. We also show the robust generalizability of AdaLRS across varying training scenarios, such as different model sizes, training paradigms, base learning rate scheduler choices, and hyperparameter settings.

Robotic-assisted rehabilitation therapy has been a promising way in improving upper limb motor function. This paper proposes a multi-mode training control method for a bilateral upper limb rehabilitation robotic system, with which human-compliant rehabilitation training can be provided. Firstly, an admittance controller is built to transform the human-robot interaction force to compliant desired trajectory. Then, by integrating with super-twisting algorithm, a nonlinear observer is designed to estimate the lumped disturbance exerted on the driving revolute joint, including the active force applied by human subject, the force of friction, the model uncertainty, et al. To guarantee that the state of position converges to the desired value in real time, a high-order sliding mode controller combined with the disturbance compensation from the observer is proposed. Additionally, based on the aforementioned several methods, multiple bilateral training modes are constructed for patients in different rehabilitation stages. The overall system including the constructed bilateral rehabilitation robotic system and the proposed control method is verified in several experiments, demonstrating the advantage of the controller on interaction compliance with respect to normal method in addition to the capability of multiple rehabilitation training modes.Note to Practitioners—This work is motivated by the patients’ needs of the compliance and comfort during the human-robot interaction in the robot-aided rehabilitation training process. Thus, a nonlinear observer-based sliding mode controller combined with admittance model is proposed in this paper. The developed control method has the following functionalities:(1)Ensuring the compliance of the desired trajectory via the constructed admittance model.(2)Solving the estimation of lumped disturbance exerted on the robotic system based on nonlinear force observer.(3)Ensuring the trajectory tracking with high accuracy under unknown disturbances via sliding mode controller combined with the observer compensation. The controller can be potentially applied in lots of areas: 1) Human-robot compliant collaboration or interaction, e.g., human-robot cooperative manipulation, robotic surgery; 2) Precise motion of robotic arm under unknown disturbance.

In this article, an asymmetric integral barrier Lyapunov function (AIBLF)-based control scheme is proposed for human–robot interaction (HRI), with which robot-aided human-compliant space-constrained muscle strength training can be achieved. First, an admittance model is exploited to generate compliant desired trajectory with the input of human–robot interaction torque. Then, on the basis of the super-twisting algorithm, a nonlinear observer is built to estimate and further compensate for the lumped disturbance applied to the robotic driving joint, including the active torque from human subject, the robotic model uncertainty, the friction, etc. Finally, an AIBLF-based controller involving nonlinear observer is proposed to solve the trajectory tracking issues in addition to the general constraint of training task space, in which the AIBLF strategy is utilized to establish an asymmetric-constrained training task space with adjustable boundary effects. This approach ensures that the training environment is tailored to accommodate individual needs and preferences, promoting a safer and more comfortable training experience. The convergence of all states and stability analysis for the closed-loop system are presented via the Lyapunov stability theory. The effectiveness of the proposed control scheme is verified by a single-joint muscle strength training robot in various experiments, and it is worth noting that this method can be easily extended to other multijoint robotic systems with the demand of human compliance and space constraint.

This work addresses the interaction control problem of a fully actuated aerial vehicle considered as a flying end-effector. We tackle the problem using geometrically consistent variable-stiffness impedance control for safe wrench regulation using the concept of energy tanks, where both the modeling and the control are carried out in the port Hamiltonian framework. We exploit previous well-known results in the literature of ground manipulators and extend them to be applied for novel and challenging aerial physical interaction with a focus on quasi-static applications. The energy-awareness of the presented control method guarantees the stability of the aerial robot in both free-flight and in-contact scenarios together with a level of safety in the case of contact-loss with the unknown environment. Furthermore, by utilizing bond graphs we demonstrate how the closed-loop passivity can be graphically conducted. The validity of our proposed approach is shown via several experiments. We also provide several insights on how the proposed framework could be extended to a generic dynamic aerial physical interaction.

The integration of computer vision techniques for the accomplishment of autonomous interaction tasks represents a challenging research direction in the context of aerial robotics. In this paper, we consider the problem of contact-based inspection of a textured target of unknown geometry and pose. Exploiting state of the art techniques in computer graphics, tuned and improved for the task at hand, we designed a framework for the projection of a desired trajectory for the robot end-effector on a generically-shaped surface to be inspected. Combining these results with previous work on energy-based interaction control, we are laying the basis of what we call vision-based impedance control paradigm. To demonstrate the feasibility and the effectiveness of our methodology, we present the results of both realistic ROS/Gazebo simulations and preliminary experiments with a fully-actuated hexarotor interacting with heterogeneous curved surfaces whose geometric description is not available a priori, provided that enough visual features on the target are naturally or artificially available to allow the integration of localization and mapping algorithms.