With the advances of deep learning, efficient model inference is crucial. Deep learning compilers optimize inference by decomposing models into subgraphs and searching schedules for them, whose evaluation relies on accurate cost models. Existing methods suffer from high transformation overheads or limited prediction accuracy caused by insufficient structural representation of subgraphs and schedules. To address these limitations, we propose CAST, a framework that predicts schedule performance based on Abstract Syntax Trees (ASTs). CAST proposes AST classification based on structural similarity and class-specific cost models. Experiments show CAST achieves significantly reduced prediction errors and up to$13 \times$higher efficiency than prior methods.

With the widespread application of deep neural networks, improving inference efficiency has become increasingly critical. To speed up the inference, deep learning compilers search for high-performance schedules for the DNN tensor programs. During the process, cost models have been extensively studied to evaluate the performance of the schedules, such that high-performance ones can be efficiently obtained. However, existing methods suffer from either high overhead or low accuracy of performance evaluation, both of which limit the efficiency of the final schedule. To address these issues, we propose FASP, a fast and accurate framework for schedule performance evaluation, based on Abstract Syntax Trees (ASTs). First, we propose a redundancy-aware ASTs reduction method to generate compact ASTs for more accurate feature extraction. Second, we propose a feature extraction method based on compact ASTs, which extracts features by accounting for computation nodes, loop nodes, and their structural relationships. Third, we propose a composition-similarity-driven ASTs classification method and a class-specific cost model architecture for more accurate performance evaluation. FASP overcomes the limitations of prior methods by significantly reducing evaluation errors. Experiments show its excellent performance in both single-model and cross-model evaluation, with errors ranging from 6 % to$\mathbf{1 3 \%}$. Moreover, FASP can obtain high-performance schedules with$13 \times$lower latency.

Cooperative inference for deep neural networks (DNNs) across edge devices has received increasing attention, due to the benefits of low latency, low power consumption, and privacy preservation. Cooperative inference partitions a DNN model into multiple segments, which will then be allocated to distributed devices for parallel inference. Nonetheless, prior works fail to comprehensively study the impact of layer configurations, dynamic network bandwidths, and heterogeneous device capabilities on the inference speed, resulting in suboptimal inference performance. In this work, we conduct a comprehensive analysis of these key factors and figure out the limitations of conventional transfer-based and redundant computation-based methods. Based on the analysis, we first propose a latency prediction agent that accounts for the layer configurations, network bandwidths, and device computing capabilities, aiming to quickly evaluate the inference latency. Furthermore, we propose RACI, a resource-aware cooperative DNNs inference framework on heterogeneous edge devices. It co-trains a model agent for model partition and a workload agent for workload allocation to generate co-optimized model partition and workload allocation strategies, leading to high cooperation inference acceleration. Experimental results demonstrate that RACI outperforms the state-of-the-art approaches by 1.1× -5.2× in terms of inference speedup for three representative DNN models.

A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic-related reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1\% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models.

The completeness of reconstructed models in low-textured areas in multi-view stereo is still a challenging problem because of the unreliable photometric consistency. Since these areas always exhibit planar properties, many methods explicitly construct planar priors to assist in optimizing depth estimation. However, the planar models they constructed are not robust enough, and the plane parameters for the same region are not consistent in different views. In this paper, we develop a novel framework to obtain high-quality planar priors. Specifically, we first propose an efficient credible point selection method. We then combine neighboring views to generate a sparse point cloud, utilize multi-planes detection and produce planar prior for each view. Therefore different views can obtain the same plane information of the considered low-textured regions in our method. Finally, we embed our novel planar prior into PatchMatch MVS to get the final depth maps. Experiments on the ETH3D datasets show our method reconstruct 3D model in low-textured areas depth estimation effectively in untextured areas.

Depth estimation is solved as a regression or classification problem in existing learning-based multi-view stereo methods. Although these two representations have recently demonstrated their excellent performance, they still have apparent shortcomings, e.g., regression methods tend to overfit due to the indirect learning cost volume, and classification methods cannot directly infer the exact depth due to its discrete prediction. In this paper, we propose a novel representation, termed Unification, to unify the advantages of regression and classification. It can directly constrain the cost volume like classification methods, but also realize the sub-pixel depth prediction like regression methods. To excavate the potential of unification, we design a new loss function named Unified Focal Loss, which is more uniform and reasonable to combat the challenge of sample imbalance. Combining these two unburdened modules, we present a coarse-to-fine framework, that we call UniMVSNet. The results of ranking first on both DTU and Tanks and Temples benchmarks verify that our model not only performs the best but also has the best generalization ability.

In this paper, we propose a method to evaluate the throughput of video transcoding in cloud services. This method can quickly estimate the maximum number of transcoding video concurrence on the current transcoding unit.

This paper presents a novel method to train the quantized feature with the action recognition task jointly. A quantization and inverse-quantization layers are introduced to the 3D CNN. The quantization and the action recognition loss functions are minimized jointly. That is, the method aims to learn the feature not only to improve action recognition accuracy but also reduce the information loss of the quantization. The framework is shown in Fig. (1).

The software x264 is an open source encoder for video compression which was released in 2003 and has been dominating the industry for the past decade. Over the last few years, an open source encoder x265 for High Efficiency Video Coding (HEVC) video coding standard was released, and it claims significant improvement over the x264. Similar to x265, an efficient video encoder namely uavs3e was designed by our team and other corporate contributors in 2020. The uavs3e software library and application are available in GitHub [1]. The main goal of uavs3e development is to achieve substantial compression gain over state-of-the-art open source codecs while minimizing the computational complexity. For higher compression capabilities, uavs3e adopts massive new coding tools, which can increase the performance of pre-diction, transform, etc. Compared with the reference soft-ware HPM4.0, the speed level 1 of uavs3e could speed up 86.1x, while the compression performance loss is not more than 2.3%. Compared with x265 veryslow preset, the uavs3e project achieves an average of 22.7%, 46.1%, and 46.2% BD bitrate saving in luminance and chrominance components respectively.

The third generation of Audio Video Coding Standard (AVS3) achieves 22% coding performance improvement compared with High Efficiency Video Coding (HEVC). However, the improvement of encoding efficiency comes from a more flexible block partition scheme is at the cost of much higher encoding complexity. This paper proposes a bottom-up fast algorithm to prune the time-consuming search process of the CU partition tree. To be specific, we design a scoring mechanism based on the splitting patterns traced back from the bottom to predict the possibility of a partition type to be selected as optimal. The score threshold to skip the exhaustive Rate-Distortion Optimization (RDO) procedure of the partition type is determined by statistical analysis. The experimental results show that the proposed methods can achieve 24.56% time-saving with 0.37% BDBR loss under Random Access configuration and 12.50% complexity reduction with 0.08% BDBR loss under All Intra configuration. The effectiveness leads to the adoption by the open-source platform of AVS3 after evaluated by the AVS working group.

MPEG has produced standards that have provided the industry with the best video compression technologies. To address diverse Internet needs, MPEG issued a Call for Proposals (CfP) for Internet video coding (IVC) in July, 2011. The anticipation is that any patent declaration associated with the baseline profile of this standard will indicate that the patent owner is prepared to grant a free of charge license to an unrestricted number of applicants worldwide. Three codecs have responded to the CfP: Web video coding (WVC), video coding for browsers (VCB), and IVC. WVC is in fact the AVC baseline, and VCB uses the same coding tools as VP8. IVC has been developed in MPEG from scratch by combining well-known existing technology elements and new coding tools with royalty-free declarations. In June 2015, the IVC project was approved as ISO/IEC 14496-33 (MPEG-4 IVC). This standard can be highly beneficial for video services in the Internet domain. This paper describes the main coding tools used in IVC, and evaluates its objective and subjective performances compared with WVC, VCB, and AVC high profile (AVC HP). The experimental results show that IVC's compression performance is approximately equal to that of the AVC HP for typical operational settings, both for streaming and low-delay applications, and is superior to WVC and VCB.

To provide full panoramic video content, the transmission bandwidth and playing complexity of virtual reality (VR) video are much higher than that of traditional video. Dynamic stream switching is an efficient scheme which stores multiple asymmetric projections of panoramic video in server side, and transmits only one asymmetric projection (with much lower resolution than full panoramic video) according to the direction of user's head. In this paper, we propose an asymmetric circular (ASC) projection method to generate the asymmetric projection of panoramic video. ASC uses equal area projection in the main viewpoint area to ensure the high and uniform quality of this area, and a projection with decreasing sampling density in other areas. Experimental results show that the bit rate and playing complexity of VR video are greatly reduced with the proposed ASC method. ASC is superior to state-of-the-art method and has been adopted in the standard of IEEE 1857.9-Immersive Visual Content Coding.

Panoramic video provides an immersive experience by presenting a 360° spherical video content. Due to the limitations of coding and storage technology, the spherical panoramic video needs to be projected onto the two-dimensional plane for storage and encoding. In this paper, we propose a polar square projection scheme. We project the area near the poles of the sphere into two square planes and a latitude circle on sphere is projected to a square circle on squares plane, in addition, the rest of area on sphere is projected into a rectangle by means of equal area projection. Experimental results show our proposed projection can obtain a gain of 11.63% BD-rate compared to the equirectangular projection.

The second generation of the audio video coding standard (AVS2), which has been issued as IEEE 1857.4, doubles the coding efficiency of AVS1 and H.264/AVC. However, the advanced techniques in AVS2 also dramatically increase the computational complexity. The research on a commercial encoder for AVS2 is still at an early stage. In this paper, we propose the first fast intra-encoding platform for AVS2, which we term iAVS2. The platform uses numerous speedup methods, including code optimization, single-instruction multiple-data acceleration, and fast algorithms for almost all the time-consuming modules. To meet different application scenarios, five different speed levels are designed in iAVS2 to provide flexible tradeoffs between compression efficiency and encoding speed. The experimental results show that the fastest speed level of iAVS2 can accelerate the AVS2 reference software (RD12.0) by 144× on average. Compared with the well-known HEVC encoder platform ×265, iAVS2 can provide over 10% bit rate saving at a similar coding speed. For the UHD and 1080 sequences, iAVS2 outperforms x265 at all speed levels.

The image-domain-warping (IDW) method can effectively create high-quality virtual views. However, the IDW algorithm is very complex, and the software implementation for this method is far from real-time. In this paper, we propose an IDW-based view synthesis acceleration method on general-purpose computing on a graphics processing unit (GPGPU). Our method makes two main contributions. First, at the algorithm level, we employ FAST for sparse disparity estimation and adopt the successive over-relaxation iterative method to calculate warps. Second, at the platform level, two computation-intensive modules (data extraction and view synthesis) in IDW are offloaded to GPU using efficient data-level parallelism strategies. Experimental results demonstrate that our proposed acceleration method can speed up the original IDW algorithm by more than 110x, and HD stereo three-dimensional video can be converted to 8-view 4 K video (each view has an approximate 720P resolution) in real-time on a hybrid CPU + GPU (NVIDIA GTX980) platform.

In high efficiency video coding (HEVC), the encoder employs a flexible quad-tree coding structure as well as a large number of prediction modes. For each size of coding unit (CU), transform unit (TU) and each prediction mode, the rate distortion optimization (RDO) is performed to select the best CU, TU and the best prediction mode. Although better coding efficiency is achieved, the computational complexity increases dramatically. In order to reduce the burden of RDO in the HEVC intra encoder, in this paper, we propose an effective approach, which is based on the generalized Gaussian distribution (GGD) model, to estimate the block level bit-rate. The weaknesses of the conventional GGD model are analyzed and relevant improvements are exploited. Our experiments show that, compared with the original RDO procedure in HM16.0, the proposed algorithm reduces RDO time by 37.7% with 0.64% BD-rate loss.

Existing view interpolation methods like DIBR require accurate disparity map which greatly limits their applications. To this aim, we propose a novel mesh-based view interpolation algorithm capable of synthesizing visually coherent virtual views with rough disparity map estimated by stereo matching algorithms. We adopt an edge-aware mesh cutting method to explicitly handle occlusion and preserve sharp depth discontinuities. Experiments on Middlebury dataset and 3D-HEVC test sequences demonstrate that proposed method outperforms DIBR and state-of-the-art mesh-based view interpolation algorithm in terms of visual quality and PSNR.

In this paper, we propose a hybrid parallel decoding strategy for HEVC which combines task-level parallelism and datalevel parallelism based on CTUs. The data-level parallelism makes the execution time distribution of different decoding stages more balanced, and makes the task-level parallelism more efficient. Our approach imposes no constraint on bit streams that they shall be generated by optional parallel coding tools such as tiles or WPP, so it can be applied for all kinds of HEVC bit streams. Furthermore, SSE, a typical SIMD instruction set on X86 platform, is utilized to accelerate time-consuming modules, which shortens the execution time gaps between different stages and make them in favor of parallel processing. We have implemented these acceleration strategies on HM-10.0 decoder, and a great speed-up ratio is achieved.

Recent cost-volume filtering-based local stereo methods have achieved comparable accuracy with global methods. However, there are still some significant outliers existing in the final disparity map. In this paper, we propose a cost-volume filtering-based local stereo matching method that employs a new combined cost and a novel secondary disparity refinement mechanism. The combined cost is formulated by a modified color census transform, truncated absolute differences of color and gradients. Symmetric guided filter is used for the cost aggregation. Different from traditional stereo matching, a novel secondary disparity refinement is proposed to further remove remaining outliers. Experimental results on Mid-dlebury benchmark show that our method ranks the 5thout of the 144 submitted methods, and is the best cost-volume filtering-based local method. Furthermore, experiments on real world sequences also validate the effectiveness of our proposed method.

Nowadays, mobile devices are capable of displaying video up to HD resolution. In this paper, we propose two acceleration strategies for Audio Video coding Standard (AVS) software decoder on multi-core ARM NEON platform. Firstly, data level parallelism is utilized to effectively use the SIMD capability of NEON and key modules are redesigned to make them SIMD friendly. Secondly, a macroblock level wavefront parallelism is designed based on the decoding dependencies among macroblocks to utilize the processing capability of multiple cores. Experiment results show that AVS (IEEE 1857) HD video stream can be decoded in real-time by applying the proposed two acceleration strategies.

MB-level parallelism is widely used in parallel video coding thanks to its merits of low latency, no performance loss and high degree of parallelism. Most of video encoders with MB-level parallelism employ MB Row Scheduling (MRS) scheme. In software video encoder, early terminate algorithms tend to cause significant difference in coding time of different MBs. Consequently, the running speeds of multiple threads are unbalanced. When the number of threads is more than that of physical cores, the running speed unbalance is further worsened by computation resources competition among multiple threads. The computation resources of multiple cores can't be fully utilized without careful handling of the above running speed unbalance. Additionally, synchronization of multiple threads can also penalize the running speed of the whole video encoder. In this paper, we analyze the running speed unbalance of multiple threads in MRS scheme, and propose a new Dynamic MB-level Scheduling (DMS) scheme for parallel video coding. DMS alleviates both the running speed unbalance and synchronization delay among multiple threads on multi-core platform. Experiment results verified that video encoder with MRS can be accelerated in average 9% by our proposed DMS, when processors are fully utilized.

A Depth Template based 2D-to-3D Video Conversion and Coding system (DTVCC) is proposed by this paper. In DTVCC, triangle meshes are exploited to describe depth template for scenes in 2D video, an interactive system is designed to generate reliable depth template, depth map can be automatically reconstructed based on the depth template and 2D video frame pixels information, and the generated 3D video is compressed just by coding the 2D video plus depth template. Experiment results show that not only high quality 3D video is generated, but also the bit rate of coding the converted 3D video is saved by 12%~38% with our proposed system of DTVCC.