Abstract Large-scale vision-language models demonstrate strong multimodal alignment and generalization across diverse tasks. Among them, CLIP stands out as one of the most successful approaches. In this work, we extend the application of CLIP to sound source localization, proposing a self-supervised method operates without explicit text input. We introduce a framework that maps audios into tokens compatible with CLIP’s text encoder, producing audio-driven embeddings. These embeddings are used to generate sounding region masks, from which visual features are extracted and aligned with the audio embeddings through a contrastive audio-visual correspondence objective. Our findings show that alignment knowledge of pre-trained multimodal foundation model enables our method to generate more complete and compact localization for sounding objects. We further propose an LLM-guided extension that distills object-aware audio-visual scene understanding into the model during training to enhance alignment. Extensive experiments across five diverse tasks demonstrate that our method, in all variants, outperforms state-of-the-art approaches and achieves strong generalization in zero-shot settings.

Neural speech and audio codecs have demonstrated decent quality of the decoded audio at low bitrates. They consist of three parts, an encoder, a decoder, and a quantizer. Residual vector quantization (RVQ) or multi-stage vector quantization in which the residual signal from the previous stage is quantized in the next stage is employed in many neural speech codecs and has exhibited good performance while providing bitrate scalability. In this letter, we propose the redundancy-reduced residual vector quantization (R3VQ) which improves the RVQ by inserting a neural network called a refiner. The role of the refiner is to reduce the power of the residual signal to be quantized by enhancing the estimate of the original speech from the quantized signals in the previous stages. We also present a part-wise (PW) training scheme suitable for the training of the neural speech codec with the R3VQ. Experimental results showed that the proposed R3VQ trained with a PW training scheme outperformed the RVQ in both objective measures for speech quality and subjective MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) test.

Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment. We extend the application of these models, specifically CLIP, to the domain of sound source localization. Unlike conventional approaches, we employ the pre-trained CLIP model without explicit text input, relying solely on the audio-visual correspondence. To this end, we introduce a framework that translates audio signals into tokens compatible with CLIP’s text encoder, yielding audio-driven embeddings. By directly using these embeddings, our method generates audio-grounded masks for the provided audio, extracts audio-grounded image features from the highlighted regions, and aligns them with the audio-driven embeddings using the audio-visual correspondence objective. Our findings suggest that utilizing pre-trained image-text models enable our model to generate more complete and compact localization maps for the sounding objects. Extensive experiments show that our method outperforms state-of-the-art approaches by a significant margin.

The goal of this work is to localize sound sources in visual scenes with a self-supervised approach. Contrastive learning in the context of sound source localization leverages the natural correspondence between audio and visual signals where the audio-visual pairs from the same source are assumed as positive, while randomly selected pairs are negatives. However, this approach brings in noisy correspondences; for example, positive audio and visual pair signals that may be unrelated to each other, or negative pairs that may contain semantically similar samples to the positive one. Our key contribution in this work is to show that using a less strict decision boundary in contrastive learning can alleviate the effect of noisy correspondences in sound source localization. We propose a simple yet effective approach by slightly modifying the contrastive loss with a negative margin. Extensive experimental results show that our approach gives on-par or better performance than the state-of-the-art methods. Furthermore, we demonstrate that the introduction of a negative margin to existing methods results in a consistent improvement in performance.

Overheating smartphones could hamper user experiences. While there have been numerous reports on smartphone overheating, a systematic measurement and user experience study on the thermal aspect of smartphones is missing. Using thermal imaging cameras, we measure and analyze the temperatures of various smartphones running diverse application workloads such as voice calling, video recording, video chatting, and 3D online gaming. Our experiments show that running popular applications such as video chat, could raise the smartphone's surface temperature to over 50$^\circ$C in only 10 minutes, which could easily cause thermal pain to users. Recent ubiquitous scenarios such as augmented reality and mobile deep learning also have considerable thermal issues. We then perform a user study to examine when the users perceive heat discomfort from the smartphones and how they react to overheating. Most of our user study participants reported considerable thermal discomfort while playing a mobile game, and that overheating disrupted interaction flows. With this in mind, we devise a smartphone surface temperature prediction model, by using only system statistics and internal sensor values. Our evaluation showed high prediction accuracy with root-mean-square errors of less than 2$^\circ$C. We discuss several insights from our findings and recommendations for user experience, OS design, and developer support for better user-thermal interactions.

We present X-Droid, a framework that provides Android app developers an ability to quickly and easily produce functional prototypes. Our work is motivated by the need for such ability and the lack of tools that provide it. Developers want to produce a functional prototype rapidly to test out potential features in real-life situations. However, current prototyping tools for mobile apps are limited to creating non-functional UI mockups that do not demonstrate actual features. With X-Droid, developers can create a new app that imports various kinds of functionality provided by other existing Android apps. In doing so, developers do not need to understand how other Android apps are implemented or need access to their source code. X-Droid provides a developer tool that enables developers to use the UIs of other Android apps and import desired functions into their prototypes. X-Droid also provides a run-time system that executes other apps' functionality in the background on off-the-shelf Android devices for seamless integration. Our evaluation shows that with the help of X-Droid, a developer imported a function from an existing Android app into a new prototype with only 51 lines of Java code, while the function itself requires 10,334 lines of Java code to implement (i.e., 200× improvement).

The SCIAMACHY instrument on the European Space Agency satellite ENVISAT measured greenhouse gases in the troposphere and stratosphere. In this study, the CH4source area is extracted by estimating the concentrations of methane emissions from time-series satellite data. Contamination of the data by cloud is interpolated both spatially and temporally. It is assumed that methane emission is negligible over ocean and that the concentration in the ocean area is due to advection from land. Background CH4concentration on land was defined as the ocean CH4concentration at the same latitude. Land CH4emission concentrations show that areas of concentrated high CH4emission are not in paddy fields only but also in broadleaf evergreen areas in South America and Central Africa.

Sharing copyright protected content without the copyright holder’s permission is illegal in many countries. Regardless, the number of illegal file sharing using BitTorrent continues to grow and most of file sharers and downloader are unconcerned legal action to transfer copywrite-protected files. However, it is difficult to gather enough probative evidence to prosecute illegal file sharers in criminal court and/or sued for damages in civil court. Further, there is a lack of research on investigation techniques to reveal illegal BitTorrent sharers. This is because the role of the server in BitTorrent networks has been changed compared to servers in conventional P2P networks. As a result, it is difficult to apply previous investigation processes for investigation of conventional P2P networks to the investigation of suspected illegal file sharing using BitTorrent. This paper proposes a methodology for the investigation of illegal file sharers using BitTorrent networks through the use of a P2P digital investigation process.

In this study, we determined the concentration of global methane (CH4) emission using SCIAMACHY data. We analyzed land and sea area to investigate the nine-year changes of CH4concentration from 2003 to 2011. Where we subtracted the CH4concentration in land and sea during 9 years. It was cleared that a big amount of CH4emission concentration was found not only in paddy fields in the Northern Hemisphere but also in broadleaf evergreen areas in the Southern Hemisphere (Central Africa and South America). We compared with the precipitation and EVI to Broadleaf Evergreen Forest and paddy field which emitted a lot of CH4. As a result, we understood that paddy field in Southeast Asia has a strong correlation with EVI and Broadleaf Evergreen Forest has a strong correlation with precipitation.

In this study, we determined the concentration of global methane (CH4) emission using SCIAMACHY data. We analyzed land and sea area to investigate the nine-year changes of CH4concentration from 2003 to 2011. Where we subtracted the CH4concentration in land and sea during 9 years. As a result, we found that the growth rate of global CH4concentration is 3-5ppb/year in land. It was cleared that a big amount of CH4emission concentration was found not only in paddy fields in the Northern Hemisphere but also in broadleaf evergreen areas in the Southern Hemisphere (Central Africa and South America).