We present an image blending pipeline, IBURD, that creates realistic synthetic images to assist in the training of deep detectors for use on underwater autonomous vehicles (AUVs) for marine debris detection tasks. Specifically, IBURD generates both images of underwater debris and their pixel-level annotations, using source images of debris objects, their annotations, and target background images of marine environments. With Poisson editing and style transfer techniques, IBURD is even able to robustly blend transparent objects into arbitrary backgrounds and automatically adjust the style of blended images using the blurriness metric of target background images. These generated images of marine debris in actual underwater backgrounds address the data scarcity and data variety problems faced by deep-learned vision algorithms in challenging underwater conditions, and can enable the use of AUVs for environmental cleanup missions. Both quantitative and robotic evaluations of IBURD demonstrate the efficacy of the proposed approach for robotic detection of marine debris.

We introduce COU: Common Objects Underwater, an instance-segmented image dataset of commonly found man-made objects in multiple aquatic and marine environments. COU contains approximately 10K segmented images, annotated from images collected during a number of underwater robot field trials in diverse locations. COU is created to address the lack of datasets with robust class coverage curated for underwater instance segmentation, which is particularly useful for training light-weight, real-time capable detectors for Autonomous Underwater Vehicles (AUVs). In addition, COU addresses the lack of diversity in object classes since the commonly available underwater image datasets focus only on marine life. Currently, COU contains images from both closed-water (pool) and open-water (lakes and oceans) environments, of 24 different classes of objects including marine debris, dive tools, and AUVs. To assess the efficacy of COU in training underwater object detectors, we use three state-of-the-art models to evaluate its performance and accuracy, using a combination of standard accuracy and efficiency metrics. The improved performance of COU-trained detectors over those solely trained on terrestrial data demonstrates the clear advantage of training with annotated underwater images. We make COU available for broad use under open-source licenses.

Underwater debris is a significantly growing challenge that autonomous underwater vehicles (AUVs) can help alleviate, but robot-guided debris search and removal can also cause harm to the aquatic ecosystem or other humans engaged in cleanup missions if the AUVs are unable to assess the risks associated with its actions. We introduce a method for identifying such risks in an underwater scene in the context of AUV debris search and removal tasks. Our approach integrates a vision language model (VLM) with monocular depth estimation to effectively classify and localize objects in a marine scene, specifically submerged marine debris. We use the pixel distance and depth difference using the monocular depth map to identify entities that are sensitive to harm in proximity to the debris. We collect and annotate a custom dataset containing images in three different marine and aquatic environments containing debris and other such sensitive entities, and compare classification performance for different types of prompts. We observe that the prompts describing the debris properties (e.g., "eroded trash") demonstrate a significant increase in accuracy compared to the use of object names directly as prompts. Our method successfully identifies debris that is safe to remove in complex scenes and turbid water conditions, highlighting the potential of using VLMs for risk assessment in AUV operations in the diverse underwater domain.

We present MeCO, the Medium Cost Open-source autonomous underwater vehicle (AUV), a versatile autonomous vehicle designed to support research and development in underwater human-robot interaction (UHRI) and marine robotics in general. An inexpensive platform to build compared to similarly-capable AUVs, the MeCO design and software are released under open-source licenses, making it a cost effective, extensible, and open platform. It is equipped with UHRI-focused systems, such as front and side facing displays, light-based communication devices, a transducer for acoustic interaction, and stereo vision, in addition to typical AUV sensing and actuation components. Additionally, MeCO is capable of real-time deep learning inference using the latest edge computing devices, while maintaining low-latency, closed-loop control through high-performance microcontrollers. MeCO is designed from the ground up for modularity in internal electronics, external payloads, and software architecture, exploiting open-source robotics and containerarization tools. We demonstrate the diverse capabilities of MeCO through simulated, closed-water, and open-water experiments. All resources necessary to build and run MeCO, including software and hardware design, have been made publicly available.

A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, Dinesh Manocha. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Toxic speech, also known as hate speech, is regarded as one of the crucial issues plaguing online social media today.Most recent work on toxic speech detection is constrained to the modality of text and written conversations with very limited work on toxicity detection from spoken utterances or using the modality of speech.In this paper, we introduce a new dataset DeToxy, the first publicly available toxicity annotated dataset for the English language.DeToxy is sourced from various openly available speech databases and consists of over 2 million utterances.We believe that our dataset would act as a benchmark for the relatively new and un-explored Spoken Language Processing task of detecting toxicity from spoken utterances and boost further research in this space.Finally, we also provide strong unimodal baselines for our dataset and compare traditional two-step and E2E approaches.Our experiments show that in the case of spoken utterances, text-based approaches are largely dependent on gold human-annotated transcripts for their performance and also suffer from the problem of keyword bias.However, the presence of speech files in DeToxy helps facilitates the development of E2E speech models which alleviate both the abovestated problems by better capturing speech clues.

Despite the demand for Computer Science (CS) related jobs that women in computing can join, there has been a decline in the number of women earning a bachelor's degree in the United States from 27% in 1997 to 19% in 2016. To reduce this gap, promote gender diversity in computing and retain students, a mentorship program was created within a computing program in a university with high research activity in the southeastern US. In this program, women undergraduate CS students are paired with corporate mentors for career guidance and support. This study focuses on understanding students' experiences, including transfer students, when participating in this mentorship program. The women transfer students' perspectives are especially critical as there are a limited number of studies highlighting their experiences, which may vary from traditional women students. In spring 2020, we implemented an IRB-approved phenomenology study. We conducted semi-structured interviews with five women undergraduate CS students, among which four were transfer students. The inductive analysis of the data shows that the mentorship program impacted students' personal as well as professional spaces. Within each of these spaces, students are impacted by the program, their mentors, and their peers in the program. The program impacts are presented here.