The composition of soundtracks for videos is a highly integrated process that synthesizes artistic creativity with technical expertise. It involves the precise synchronization of musical elements with the visual component to effectively enhance the artistic quality of the video. Herein, we aimed to generate harmonious and temporally aligned music based on video content, assisting artists in soundtrack composition. Existing approaches predominantly generate music based on human motion data from fixed-length videos, limiting their applicability. To overcome this limitation, we propose an autoregressive unit diffusion model,$\mathbf {W^{2}}$, for variable-length video-to-music generation.$\mathbf {W^{2}}$segments videos and music into unified temporal units and employs a diffusion model to generate music units conditioned on structured video information and contextual music information. It then autoregressively generates coherent and consistent music sequences. This model employs structured video condition information, obtained through global encoding, to effectively generate music units that consistently align with the video content. At the same time, contextual music condition information is used to ensure the generation of a continuous sequence of music units matching the duration of the video. Unlike prior models relying on human skeletal features,$\mathbf {W^{2}}$can be applied to not only dance videos but also free-style videos. Experimental results show that$\mathbf {W^{2}}$generates music that is temporally synchronized and semantically aligned with visual content.

For the scenes of stage lighting design in entertainment concerts, this paper proposes a DMX-based generative model, LightingGen, to simulate the lighting design and programming process of human experts. Unlike previous emotion-based lighting design approaches, LightingGen employs a data-driven method from the creator's perspective, standardizing stage lighting control parameters through DMX signals to create a unified lighting feature representation called DULA. LightingGen incorporates a two-stage generative model: the first stage uses LightingCodec to generate latent representations of lighting features, and the second stage uses music as a condition to perform conditional sampling in the latent spaces of lighting. To validate our methods, A virtual stage environment was built to collect 2.62 hours of DMX lighting control sequences as dataset. Experimental results indicate that LightingGen can generate lighting control parameters that partially align with pop music. To our knowledge, LightingGen is the first work to use generative methods to model the relationship between pop music and stage lighting.