Combining deep learning and bird sound recognition strongly supports monitoring bird species and maintaining ecological balance. However, in outdoor environments, the extraction of bird sound features is often hindered by environmental noise, making it challenging for models to learn the fine-grained features of bird sounds fully. And single-scale feature extraction is harrowing to cover the time–frequency domain feature information of bird sounds in multiple dimensions. To address these issues, this paper proposes a multi-grained detail-enhanced and patch-aware network. The model utilizes densely connected time delay neural network as the backbone network and introduces the multi-grained detail-enhanced convolution, which combines vanilla convolutions with differential convolutions in the horizontal, vertical, angular, and central levels, and incorporates multi-grained pooling strategies to learn fine-grained acoustic features at different levels. To further overcome the limitations of single-scale feature extraction, the branch patch-aware attention module is proposed. This module collaboratively captures local details and global contextual information through a multi-branch structure and patch partitioning of different sizes. On the three datasets, the method achieved accuracies of 96.29%, 86.51%, and 97.40%, respectively. This achievement demonstrates the precise capture and parsing ability of the method for audio feature information.

3D object detection (OD) is a crucial element in scene understanding. However, most existing 3D OD models have been tailored to work with light detection and ranging (LiDAR) and RGB-D point cloud data, leaving their performance on commonly available visual-inertial simultaneous localization and mapping (VI-SLAM) point clouds unexamined. In this paper, we create and release two datasets: VIP500, 4772 VI-SLAM point clouds covering 500 different object and environment configurations, and VIP500-D, an accompanying set of 20 RGB-D point clouds for the object classes and shapes in VIP500. We then use these datasets to quantify the differences between VI-SLAM point clouds and dense RGB-D point clouds, as well as the discrepancies between VI-SLAM point clouds generated with different object and environment characteristics. Finally, we evaluate the performance of three leading OD models on the diverse data in our VIP500 dataset, revealing the promise of OD models trained on VI-SLAM data; we examine the extent to which both object and environment characteristics impact performance, along with the underlying causes.

Object detection (OD) is crucial for numerous emerging visual sensing applications. As OD models trained on unrepresentative data usually yield poor performance, collecting high-quality data in the local environment is recognized to be essential for improving model accuracy. Yet, the question of how to collect this data is currently largely overlooked; unsupported data collection tends to produce datasets with a significant proportion of redundant or uninformative data, hindering effective model training. To address this challenge, we design a real-time data importance estimation method and integrate it into BiGuide, a bi-level image data acquisition system we create for OD tasks. BiGuide assesses the importance of the captured images in real-time based on informativeness and diversity estimations and dynamically guides users in collecting useful data via image-level and object instance-level guidance. We prototype BiGuide in an edge-based architecture using commodity smartphones as mobile clients, and evaluate its performance via an IRB-approved study with 20 users. Our evaluation demonstrates that OD models trained on the data collected by BiGuide outperform models trained on the data collected by two baseline systems, achieving detection accuracy improvements of up to 33.07% and 14.57%, respectively. Over 85% of the users found BiGuide fast, helpful, and easy to understand and follow.

Edge-cloud collaborative inference empowers resource-limited IoT devices to support deep learning applications without disclosing their raw data to the cloud server, thus protecting user's data. Nevertheless, prior research has shown that collaborative inference still results in the exposure of input and predictions from edge devices. To defend against such data leakage in collaborative inference, we introduce InfoScissors, a defense strategy designed to reduce the mutual information between a model's intermediate outcomes and the device's input and predictions. We evaluate our defense on several datasets in the context of diverse attacks. Besides the empirical comparison, we provide a theoretical analysis of the inadequacies of recent defense strategies that also utilize mutual information, particularly focusing on those based on the Variational Information Bottleneck (VIB) approach. We illustrate the superiority of our method and offer a theoretical analysis of it.

Real-time object detection (OD) is a key enabling technology for a wide range of emerging mobile system applications. However, deploying an OD model pre-trained on a public dataset (source domain) in a specific local environment (target domain) is known to lead to significant performance degradation because of the so-called domain gap between the dataset and the environment. Collecting local data and fine-tuning the OD model on this data is a commonly used approach for improving the robustness of OD models in real-world deployments. Yet, the question of how to collect this data is currently largely overlooked; unsupported data collection is likely to produce datasets that contain significant proportion of redundant or uninformative data for model training. In this demo, we present BiGuide, a bi-level image data acquisition guidance for OD tasks, to guide users to change their camera locations or angles to different extents (significantly or slightly) to obtain the data which benefits model training via image-level and object instance-level guidance. We showcase an interactive demonstration of collecting data for a lemur species detection application we are developing and deploying at the Duke Lemur Center. Demo participants will take pictures of lemur toys with the mobile phone under the real-time guidance and will observe the real-time display of the metrics that assess the importance of the captured data. They will develop an intuition for how real-time image importance assessment and bi-level guidance improve the quality of collected data.

Federated learning (FL) has attracted increasing attention as a promising technique to drive a vast number of edge devices with artificial intelligence. However, it is very challenging to guarantee the efficiency of a FL system in practice due to the heterogeneous computation resources on different devices. To improve the efficiency of FL systems in the real world, asynchronous FL (AFL) and semi-asynchronous FL (SAFL) methods are proposed such that the server does not need to wait for stragglers. However, existing AFL and SAFL systems suffer from poor accuracy and low efficiency in realistic settings where the data is non-IID distributed across devices and the on-device resources are extremely heterogeneous. In this work, we propose FedSEA - a semi-asynchronous FL framework for extremely heterogeneous devices. We theoretically disclose that the unbalanced aggregation frequency is a root cause of accuracy drop in SAFL. Based on this analysis, we design a training configuration scheduler to balance the aggregation frequency of devices such that the accuracy can be improved. To improve the efficiency of the system in realistic settings where the devices have dynamic on-device resource availability, we design a scheduler that can efficiently predict the arriving time of local updates from devices and adjust the synchronization time point according to the devices' predicted arriving time. We also consider the extremely heterogeneous settings where there exist extremely lagging devices that take hundreds of times as long as the training time of the other devices. In the real world, there might be even some extreme stragglers which are not capable of training the global model. To enable these devices to join in training without impairing the systematic efficiency, Fed-SEA enables these extreme stragglers to conduct local training on much smaller models. Our experiments show that compared with status quo approaches, FedSEA improves the inference accuracy by 44.34% and reduces the systematic time cost and local training time cost by 87.02× and 792.9×. FedSEA also reduces the energy consumption of the devices with extremely limited resources by 752.9×.

Scene classification in very high-resolution (VHR) remote sensing (RS) images is a challenging task due to the complex and diverse content of the images. Recently, convolution neural networks (CNNs) have been utilized to tackle this task. However, CNNs cannot fully meet the needs of scene classification due to clutters and small objects in VHR images. To handle these challenges, this letter presents a novel multilevel feature fusion (MLFF) network with adaptive channel dimensionality reduction for RS scene classification. Specifically, an adaptive method is designed for channel dimensionality reduction of high-dimensional features. Then, an MLFF module is introduced to fuse the features in an efficient way. Experiments on three widely used data sets show that our model outperforms several state-of-the-art methods in terms of both accuracy and stability.

In this paper, we propose a new dataset and benchmark for low altitude UAV object detection, aiming to find and localize waste plastic bottles in the wild, as well as to inspire the development of object detection models to be capable of detecting small and transparent objects. To this end, we collect 25, 407 UAV images of bottles with various kinds of backgrounds. Unlike traditional horizontal bounding box based annotation methods, we use the oriented bounding box to accurately and compactly annotate the bottles, which provides more detailed information for subsequent robotic grasping. The fully annotated images contain 34, 791 bottles, each of which is annotated by an arbitrary (5 d.o.f.) quadrilateral. To build a baseline for bottle detection, we evaluate several state-of-the-art object detection algorithms on our UAV-Bottle Dataset (UAV-BD), such as Faster R-CNN, SSD, YOLOv2 and RRPN. We also present an analysis of the dataset along with baseline approaches. Both the dataset and benchmark are made publicly available to the vision community on our website to advance research in the area of object detection from UAVs.