Video-to-music (V2M) generation aims to create music that aligns with visual content. However, two main challenges persist in existing methods: (1) the lack of explicit rhythm modeling hinders audiovisual temporal alignments; (2) effectively integrating various visual features to condition music generation remains non-trivial. To address these issues, we propose Diff-V2M, a general V2M framework based on a hierarchical conditional diffusion model, comprising two core components: visual feature extraction and conditional music generation. For rhythm modeling, we begin by evaluating several rhythmic representations, including low-resolution mel-spectrograms, tempograms, and onset detection functions (ODF), and devise a rhythmic predictor to infer them directly from videos. To ensure contextual and affective coherence, we also extract semantic and emotional features. All features are incorporated into the generator via a hierarchical cross-attention mechanism, where emotional features shape the affective tone via the first layer, while semantic and rhythmic features are fused in the second cross-attention layer. To enhance feature integration, we introduce timestep-aware fusion strategies, including feature-wise linear modulation (FiLM) and weighted fusion, allowing the model to adaptively balance semantic and rhythmic cues throughout the diffusion process. Extensive experiments identify low-resolution ODF as a more effective signal for modeling musical rhythm and demonstrate that Diff-V2M outperforms existing models on both in-domain and out-of-domain datasets, achieving state-of-the-art performance in terms of objective metrics and subjective comparisons.

Lyric-to-melody generation aims to automatically create melodies based on given lyrics, requiring the capture of complex and subtle correlations between them. However, previous works usually suffer from two main challenges: 1) lyric-melody alignment modeling, which is often simplified to one-syllable/word-to-one-note alignment, while others have the problem of low alignment accuracy; 2) lyric-melody harmony modeling, which usually relies heavily on intermediates or strict rules, limiting model's capabilities and generative diversity. In this paper, we propose SongGLM, a lyric-to-melody generation system that leverages 2D alignment encoding and multi-task pre-training based on the General Language Model (GLM) to guarantee the alignment and harmony between lyrics and melodies. Specifically, 1) we introduce a unified symbolic song representation for lyrics and melodies with word-level and phrase-level (2D) alignment encoding to capture the lyric-melody alignment; 2) we design a multi-task pre-training framework with hierarchical blank infilling objectives (n-gram, phrase, and long span), and incorporate lyric-melody relationships into the extraction of harmonized n-grams to ensure the lyric-melody harmony. We also construct a large-scale lyric-melody paired dataset comprising over 200,000 English song pieces for pre-training and fine-tuning. The objective and subjective results indicate that SongGLM can generate melodies from lyrics with significant improvements in both alignment and harmony, outperforming all the previous baseline methods.

Aggressive vehicle maneuvers such as those performed by professional racing drivers achieve high agility motion at the edge of handling limits. These aggressive maneuvers can be used to design human-inspired active safety features for next-generation “accident-free” vehicles. We present a motion planning and control design for autonomous aggressive vehicle maneuvers. The motion planner takes advantages of the sparse stable trees and the enhanced rapidly exploring random tree (RRT*) algorithms. The use of the sparsity property helps to reduce the computational cost of the RRT* method by removing non-useful nodes in each iteration and therefore to rapidly converge to the optimal solution. The proposed motion control design allows the vehicle to operate outside the stability region to accomplish a safe, agile maneuver. A safety region is computed to augment the stability region and the motion control is built on a modified nonlinear model predictive control method. We implement the proposed planner and controller and demonstrate the autonomous aggressive maneuvers on a 1/7-scale racing vehicle platform. Comparison with human expert driver and other existing methods is also presented to demonstrate the performance and robustness.Note to Practitioners—Motion planning and control of human driver-inspired aggressive vehicle maneuvers is a challenging task because of high-agility, unstable fast vehicle motions. This paper is motivated by addressing this challenge in autonomous driving technologies. Instead of restricting vehicle motions within a stability region that is taken by existing methods, we augment the conservative stability region to a safety region with guaranteed performance. To improve the computational efficiency of sampling-based motion planners, we take advantage of sparsity and also integration of a nonlinear predictive control method to compute feasible vehicle motion in searching space. The stability of the vehicle motion controller and sub-optimality of the motion planner are analyzed and guaranteed. Using a scaled vehicle testbed, we validate and compare the proposed motion planning and control design with other existing methods and human expert driver. The experimental results demonstrate the superior performance than the other methods and comparable with human expert driving skills.