For autonomous vehicles (AVs), the ability for effective end-to-end perception and future trajectory prediction is critical in planning a safe automatic maneuver. In the current AVs systems, perception and prediction are two separate modules. The prediction module receives only a restricted amount of information from the perception module. Furthermore, perception errors will propagate into the prediction module, ultimately having a negative impact on the accuracy of the prediction results. In this paper, we present a novel framework termed BEV-TP, a visual context-guided center-based transformer network for joint 3D perception and trajectory prediction. BEV-TP exploits visual information from consecutive multi-view images and context information from HD semantic maps, to predict better objects’ centers whose locations are then used to query visual features and context features via the attention mechanism. Generated agent queries and map queries facilitate learning of the transformer module for further feature aggregation. Finally, multiple regression heads are used to perform 3D bounding box detection and future velocity prediction. This center-based approach achieves a differentiable, simple, and efficient E2E trajectory prediction framework. Extensive experiments conducted on the nuScenes dataset demonstrate the effectiveness of BEV-TP over traditional pipelines with sequential paradigms.

In recent years, the design of models for performing the trajectory prediction task, one of the critical tasks in autonomous driving, has received great attention from researchers. However, accurately predicting future locations is challenging due to the difficulty of learning accurate intentions and modeling multimodality. Historical paths at a certain location can help predict the future trajectory of an agent currently located in that position and address these limitations. In this work, we propose a goal-based local behavior guided model, Goal-LBP, using such information (referred to as local behavior data) to generate potential goals and guide the prediction of trajectories conditioned on such goals. Goal-LBP uses Transformer encoders to extract homogeneous features and attention mechanism to represent the heterogeneous interactions and subsequently uses an encoder-decoder Gated Recurrent Unit (GRU) model to generate predictions. We evaluate our Goal-LBP using two large-scale real-world autonomous driving datasets, namely nuScenes and Argoverse. Our results show that compared to several SOTA models, Goal-LBP achieves the best ADE/FDE performance and it ranked #2 on the leaderboard of the nuScenes trajectory benchmark in June 2023. In addition, we also demonstrate that our local behavior estimator block can be easily added to two existing SOTA methods, namely AgentFormer and LaPred. Adding this LBE block improves the original AgentFormer and LaPred performance by at least 10%.

The task of instance segmentation in videos aims to consistently identify objects at pixel level throughout the entire video sequence. Existing state-of-the-art methods either follow the tracking-by-detection paradigm to employ multi-stage pipelines or directly train a complex deep model to process the entire video clips as 3D volumes. However, these methods are typically slow and resource-consuming such that they are often limited to offline processing. In this paper, we propose SRNet, a simple and efficient framework for joint segmentation and tracking of object instances in videos. The key to achieving both high efficiency and accuracy in our framework is to formulate the instance segmentation and tracking problem into a unified spatial-relation learning task where each pixel in the current frame relates to its object center, and each object center relates to its location in the previous frame. This unified learning framework allows our framework to perform join instance segmentation and tracking through a single stage while maintaining low overheads among different learning tasks. Our proposed framework can handle two different task settings and demonstrates comparable performance with state-of-the-art methods on two different benchmarks while running significantly faster.

Training a semantic segmentation model requires large densely-annotated image datasets that are costly to obtain. Once the training is done, it is also difficult to add new object categories to such segmentation models. In this paper, we tackle the few-shot semantic segmentation problem, which aims to perform image segmentation task on unseen object categories merely based on one or a few support example(s). The key to solving this few-shot segmentation problem lies in effectively utilizing object information from support examples to separate target objects from the background in a query image. While existing methods typically generate object-level representations by averaging local features in support images, we demonstrate that such object representations are typically noisy and less distinguishing. To solve this problem, we design an object representation generator (ORG) module which can effectively aggregate local object features from support im- age(s) and produce better object-level representation. The ORG module can be embedded into the network and trained end-to-end in a weakly-supervised fashion without extra human annotation. We incorporate this design into a modified encoder-decoder network to present a powerful and efficient framework for few-shot semantic segmentation. Experimental results on the Pascal-VOC and MS-COCO datasets show that our approach achieves better performance compared to existing methods under both one-shot and five-shot settings.

Forecasting human poses given a sequence of historical pose frames has several important applications, especially in the domain of smart home safety. Recently, computer vision-based human pose forecasting has made a breakthrough using deep learning technology. However, to implement a practical system deployed on an IoT edge environment, there are still two issues to be addressed. First, existing methods on pose forecasting fail to model the coherent structural information of connected human joints and thus cannot achieve satisfactory prediction accuracy, especially for long-term predictions. Second, a general and static pre-trained prediction model may not perform well in the deployment environment due to the visual domain shift problem. In this article, we propose a hybrid cloud-edge system called GPFS to solve those issues. Specifically, we first introduce a novel graph convolutional neural network (GCN)-based sequence-to-sequence learning method, which enhances the sequence encoder by using a graph to represent both the spatial and temporal connections of the human joints in the input frames. The GCN improves the forecasting accuracy by capturing the motion pattern of each joint as well as the correlations among different human joints over time. Second, to address the domain shift issue and protect data privacy, we extend the system to perform online learning on the IoT edge to adapt the cloud trained general model with online collected on-site domain data. Extensive evaluation on Human 3.6M and Penn Action datasets demonstrates the superiority of our proposed system.

Deploying a deep learning model on mobile/IoT devices is a challenging task. The difficulty lies in the trade-off between computation speed and accuracy. A complex deep learning model with high accuracy runs slowly on resource-limited devices, while a light-weight model that runs much faster loses accuracy. In this paper, we propose a novel decomposition method, namely DAC, that is capable of factorizing an ordinary convolutional layer into two layers with much fewer parameters. DAC computes the corresponding weights for the newly generated layers directly from the weights of the original convolutional layer. Thus, no training (or fine-tuning) or any data is needed. The experimental results show that DAC reduces a large number of floating-point operations (FLOPs) while maintaining high accuracy of a pre-trained model. If 2% accuracy drop is acceptable, DAC saves 53% FLOPs of VGG16 image classification model on ImageNet dataset, 29% FLOPS of SSD300 object detection model on PASCAL VOC2007 dataset, and 46% FLOPS of a multi-person pose estimation model on Microsoft COCO dataset. Compared to other existing decomposition methods, DAC achieves better performance.

Modern face recognition systems are accurate but they are vulnerable to different types of spoofing attacks. To solve this problem, conventional face authentication systems typically employ an additional module to analyze the liveness of the input faces before feeding it into the face recognition module. Such two-stage designs not only suffer from longer processing time but also require more storage and resources, which are usually limited on mobile and embedded platforms. In this paper, we propose a multi-task Convolutional Neural Network(CNN), namely LiveFace, for face-authentication. Given an input face image, LiveFace generates two outputs through a single stage: (i) a face representation that can be used for identification or verification, and (ii) the corresponding liveness score. The two tasks share lower layers to reduce the computation cost. Experimental results using three datasets show that our model achieves a comparable performance on both face recognition and anti-spoofing tasks but much faster than conventional authentication systems. In addition, we have implemented a prototype of our scheme on Android phones and demonstrated that our scheme can run in real-time on three Android devices that we have tested.

Designing a scheme that can achieve a good performance in predicting single person activities and group activities is a challenging task. In this paper, we propose a novel robust and efficient human activity recognition scheme called ReHAR, which can be used to handle single person activities and group activities prediction. First, we generate an optical flow image for each video frame. Then, both video frames and their corresponding optical flow images are fed into a Single Frame Representation Model to generate representations. Finally, an LSTM is used to predict the final activities based on the generated representations. The whole model is trained end-to-end to allow meaningful representations to be generated for the final activity recognition. We evaluate ReHAR using two well-known datasets: the NCAA Basketball Dataset and the UCFSports Action Dataset. The experimental results show that the proposed ReHAR achieves a higher activity recognition accuracy with an order of magnitude shorter computation time compared to the state-of-the-art methods.

Activity recognition has become an important function in many emerging computer vision applications e.g. automatic video surveillance system, human-computer interaction application, and video recommendation system, etc. In this paper, we propose a novel semantics based group activity recognition scheme, namely SBGAR, which achieves higher accuracy and efficiency than existing group activity recognition methods. SBGAR consists of two stages: in stage I, we use a LSTM model to generate a caption for each video frame; in stage II, another LSTM model is trained to predict the final activity categories based on these generated captions. We evaluate SBGAR using two well-known datasets: the Collective Activity Dataset and the Volleyball Dataset. Our experimental results show that SBGAR improves the group activity recognition accuracy with shorter computation time compared to the state-of-the-art methods.

Image retrieval has become an important function in many emerging computer vision applications e.g. online shopping via images, medical health care systems. More and more images are being generated and stored in public clouds. However, recent photo leakage events raise concerns about privacy leaks for images stored in public clouds. In this paper, we present an efficient scalable hierarchical image retrieval system (CASHEIRS) which provides privacy-aware image retrieval feature. CASHEIRS employs transformed Convolutional Neural Network features to improve image retrieval accuracy and an encrypted hierarchical index tree to speed up the query process. Extensive evaluations using Caltech256 and INRIA Holiday datasets show that CASHEIRS is more effective than three existing schemes. We also demonstrate its practicality on a mobile device.