The ability to automatically build 3D digital twins of plants from images has countless applications in agriculture, environmental science, robotics, and other fields. However, current 3D reconstruction methods fail to recover complete shapes of plants due to heavy occlusion and complex geometries. In this work, we present a novel method for 3D modeling of agricultural crops based on optimizing a parametric model of plant morphology via inverse procedural modeling. Our method first estimates depth maps by fitting a neural radiance field and then optimizes a specialized loss to estimate morphological parameters that result in consistent depth renderings. The resulting 3D model is complete and biologically plausible. We validate our method on a dataset of real images of agricultural fields, and demonstrate that the reconstructed canopies can be used for a variety of monitoring and simulation applications. Project page: https://ajzhai.github.io/CropCraft

Machine learning classifier may leak sensitive information from data providers. Attackers can use an algorithm called membership inference attack (MIA) to infer whether samples have been used as training data for the classifier. To prevent MIAs, researchers have proposed various defense methods. However, these methods often struggle to balance three constraints: the trade-off between utility and privacy, the impact on prediction accuracy, and effectiveness against both NN and Metric-based MIAs. Therefore, we propose a defense strategy called DifGuard, which is based on differential privacy and uses different privacy budgets depending on data membership during the model inference phase. To validate the effectiveness of DifGuard, we conducted extensive experiments on multiple publicly available datasets. Experimental results demonstrate that, compared to the state-of-the-art defense strategies, Dif- Guard can more effectively mitigate various risks of MIA without compromising classification accuracy.

Hand detection, often as a pre-processing step for hand keypoint localization and hand segmentation, is a critical task in gesture interaction. A fast hand detector is demanding because it saves time for downstream tasks, leading to a real-time performance in hand gesture interaction. Achieving fast hand detection is challenging, especially on resource-limited devices. In this paper, we propose a lightweight real-time hand detection model called BlazeYOLO. We reduce the model's computational cost by utilizing light modules and pruning redundancy channels. A BlazeBlock, which consists of depthwise convolution, pointwise convolution, and residual structure, serves as the building block of the model backbone. The Only Train Once method trains and prunes the entire model. The model parameters are grouped into zero-invariant groups. During training, the parameters within the same group are updated or set to zero simultaneously. Due to the channel dependency induced by depthwise convolution and residual structure, we design a pruning scheme for the BlazeYOLO model. Following the pruning scheme, the groups whose parameters are all zero can be directly pruned without affecting the model output. The pruned model is the final model and does not require additional fine-tuning. Our proposed model achieves 95.5% average precision (AP) with only 0.11 GFLOPS of computational cost, reaching a high inference speed of 78.1 FPS on mobile devices.

Convolutional Neural Network (CNN) models are becoming complex with advanced OPs and structures, which introduces design challenges for FPGA-based system. In this paper, we present the design of an FPGA-based CNN inference system, PAI-FCNN, to support modern complex CNN models. PAI-FCNN consists of scalable hardware design and a model reconstruction flow in software compiler. In this way, advanced OPs like Deconv, Conv with upsampling, Dilated Conv, Concatenation can be processed by PAI-FCNN with high performance and hardware efficiency. PAI-FCNN also incorporates reduced precision to boost computing capacity, and the emerging CNN-RNN (Recurrent Neural Network) hybrid models are supported. Our experiments on both PC and embedded FPGA platforms show that the system consistently performs in an efficient manner. PAI-FCNN achieves better throughput and power efficiency than GPU solutions.

We describe the FPGA subsystem of the Platform of Artificial Intelligence (PAI) in Alibaba Group, called PAI-FCNN. PAI-FCNN plays the role of a heterogeneous back-end for CNN inference, together with other CPU, GPU and ASIC subsystems in PAI. Driven by various business needs, we built PAI-FCNN from scratch since two years ago. We present our experience from FPGA/compiler design and implementation, to system evaluation and deployment. In particular, in order to address three practical challenges: (1) Efficient processing for diverse operators and model structure such as Deconv, Dilated Conv, Up-sampling, PReLu and Concatenation. (2) Serving multiple highly-different models on single FPGA hardware. (3) Competitive performance with alternative GPU or ASIC solutions, we extensively perform joint software & hardware design to optimize system efficiency across multiple CNN models, which includes model reconstruction in compiler software and flexible data access in data-flow CNN processor. We also incorporate reduced precision and model retraining to boost system capacity. Using U-net as an example, on Xilinx KU115 chip, with the help of 74.9% efficiency on Int16-precision hardware (with 3.226TOPS capacity) and 72.9% efficiency on mixed-int8/int3-precision hardware (with 14.746TOPS capacity), we achieve slightly better throughput and 2X higher power efficiency than P4.

This article consists of a collection of slides from the author's conference presentation.