Navigational aids for blind and low vision individuals struggle conveying dynamic real-world environments, leading to cognitive overload from continuous, undifferentiated feedback. We present AMAVA, a novel real-time video-to-audio framework that converts mobile device video into contextually relevant sound effects or text-to-speech descriptions. We propose a motion-aware pipeline using a lightweight AI classification model to distinguish between low and high-movement scenes followed by a real-time text-to-audio synthesis pipeline to enhance environmental perception more efficiently. In static environments, AMAVA generates spoken audio scene descriptions for situational awareness. In high-movement situations, it prioritizes safety by delivering sound cues, such as spoken hazard alerts and environmental sound effects. These audio outputs are produced by a decoder-only transformer-based vision-language model with mixture-of-experts and cross-modal attention for visual understanding, in conjunction with neural text-to-speech and natural sound synthesis networks. The proposed framework uses prompt-based caching and category-specific throttling to avoid auditory clutter and minimize latency. We present a comprehensive evaluation of the system, including a real-time navigation study comparing a white cane alone versus with AMAVA, that shows a significant increase in user confidence and perceived safety.

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of the visual and audio features. In this paper we introduce a novel method for guiding a class-conditioned GAN to synthesize representative audio with temporally extracted visual information. We accomplish this visual-to-sound generation task by adapting the synchronicity traits between the audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading to the generation of visually aligned realistic soundtracks. We expanded our previously proposed Automatic Foley data set. We evaluated FoleyGAN's synthesized sound output through human surveys that show noteworthy (on average 81%) audio-visual synchronicity performance. Our approach outperforms other baseline models and audio-visual data sets in statistical and ablation experiments achieving improved IS, FID and NDB scores. In ablation analysis we showed the significance of our visual and temporal feature extraction method as well as augmented performance of our generation network. Overall, our FoleyGAN model showed sound retrieval accuracy of 76.08% surpassing existing visual-to-audio synthesis deep neural networks.

In movie productions, the Foley artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist identify sounds that enhance the experience for the listener, reinforcing the director's intention for the scene. The artist must decide what artificial sound captures the essence of the sound and action depicted in the scene. In this paper, we present AutoFoley, an automated deep-learning tool that is used to synthesize a representative audio track for videos. AutoFoley is used to associate audio files with soundless video or to identify critical scenarios and provide a synthesized, reinforced and time-synchronized soundtrack. Our algorithm is capable of precise recognition of actions as well as interframe relations in fast- moving video clips through incorporating interpolation technique and temporal relational networks (TRN). We employ a robust multiscale recurrent neural network (RNN) and a convolutional neural network (CNN) for better understanding of the intricate input-to-output associations. To evaluate AutoFoley, we create an audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. While the Foley dataset was limited to short-duration videos off the representative activities, this dataset demonstrates the capabilities of our proposed system. We show the synthesized sounds are portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley shows more than 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in comparable cross-modal research.