Recent advancements in array-camera videography enable real-time capturing of ultra-high-definition (Ultra-HD) videos, providing rich visual information in a large field of view. However, promptly processing such data using state-of-the-art transformer-based vision foundation models faces significant computational overhead in on-device computing or transmission overhead in cloud computing. In this paper, we present Hyperion, the first cloud-device collaborative framework that enables low-latency inference on Ultra-HD vision data using off-the-shelf vision transformers over dynamic networks. Hyperion addresses the computational and transmission bottleneck of Ultra-HD vision Transformer inference by exploiting the intrinsic property in vision Transformer models. Specifically, Hyperion integrates a collaboration-aware importance scorer that identifies critical regions at the patch level, a dynamic scheduler that adaptively adjusts patch transmission quality to balance latency and accuracy under dynamic network conditions, and a weighted ensembler that fuses edge and cloud results to improve accuracy. Experimental results on real-world prototypes and datasets demonstrate that Hyperion enhances frame processing rate by up to 1.61 × and improves the accuracy by up to 20.2% when compared with state-of-the-art baselines under various network environments.

Vision Transformers (ViTs) have outperformed traditional Convolutional Neural Network architectures and achieved state-of-the-art results in various computer vision tasks. Since ViTs are computationally expensive, the models either have to be pruned to run on resource-limited edge devices only or have to be executed on remote cloud servers after receiving the raw data transmitted over fluctuating networks. The resulting degraded performance or high latency all hinder their widespread applications. In this paper, we present Janus, the first framework for low-latency cloud-device collaborative Vision Transformer inference over dynamic networks. Janus overcomes the intrinsic model limitations of ViTs and realizes collaboratively executing ViT models on both cloud and edge devices, achieving low latency, high accuracy, and low communication overhead. Specifically, Janus judiciously combines token pruning techniques with a carefully designed fine-to-coarse model splitting policy and non-static mixed pruning policy. It attains a balance between accuracy and latency by dynamically selecting the optimal pruning level and split point. Experimental results across various tasks demonstrate that Janus enhances throughput by up to 5.15x and reduces latency violation ratios by up to 98.7% when compared with baseline approaches under various network environments.

Because of the imaging mechanism complexity of long-linear-array and wide-swath whisk-broom thermal infrared spectrometer (TIS) of the first Sustainable Development Goals Satellite (SDGSAT-1), how to achieve a high geometric positioning accuracy (GPA) becomes the core factor in subsequent geometric quantitative applications. Here, in this article, a three-step in-orbit geometric calibration (GC) strategy comprising the estimations of exterior orientation parameters (EOPs), interior orientation parameters (IOPs), and scanning compensation parameters (SCPs) is proposed to correct the geo-location displacements for whisk-broom TIS. First, in accordance with the optical-mechanical structure and pinhole imaging theory, we establish the rigorous geometric positioning model (RGPM) of TIS and analyze the error resources term-by-term along the error propagation link elaborately. Second, the corresponding rigorous geometric calibration model (RGCM) is constructed in detail based on the 2-D look-angle model and the generalized bias correction matrix. Especially for eliminating the systematic nonlinear errors in the scanning direction, a fifth-degree polynomial is put forward to be employed to fit and compensate for the angular measurement errors of the scanning mirror. Finally, a three-step estimation method is presented to estimate the calibration parameters with ground control points (GCPs). Experimental results based on the spatial references of Landsat 8 panchromatic images and version 2 of advanced spaceborne thermal emission and reflection radiometer (ASTER) global digital elevation model (GDEM2) show that the GPA of the proposed method in along-track and cross-track directions can be better than 1.0 pixels for all three bands, which makes a great sense for associated geometric measurements.

Accurate ground control points’ (GCPs) extraction is extremely essential to support the on-orbit geometric calibration and registrations of multitemporal and multispectral remote sensing images (RSIs). However, compared with other images, the thermal infrared (TIR) images responding to the targets’ temperature usually present low spatial resolution, poor contrast, and different mapping intensity, which make it difficult to get enough precision matches in the corresponding image pairs for GCPs’ extraction. Furthermore, with more attention to the gradient properties surrounding the interest points, the conventional feature-based algorithms generally neglect the plenty of geometric textural features of RSIs. Here, in this letter, we propose an accurate geometric-texture-based GCPs’ extraction approach for TIR RSIs. The novel textural log-polar pattern and the double constrained matching rules comprising the matching bits and differences are combined to guarantee the ultimate GCPs’ accuracy. The experimental results evaluated on TIR RSIs of Landsat 8 and GLS2000 show that the absolute matching errors of the proposed method in sample and line directions can be 0.50 and 0.47 pixels, which improve a lot in terms of three state-of-the-art methods.

Affected by the large and inconsistent geometric distortion deriving from long-linear-array and wide-swath whiskbroom (LWW) imaging and the small overlap area of adjacent images, the conventional stitching methods based on feature matching greatly suffer from the severe misalignment, ghosting effect and distortion in overlapping and non-overlapping areas, respectively. Here, in view of these problems, this letter proposes a resolution-normalizing stitching method for LWW images. Firstly, considering the variation of scan-angle, earth curvature, and pixel position along-track, we establish a resolution characterization model to calculate the accurate ground sampling distance (GSD) anywhere in the field of view. Then, accordingly, the resolution-normalizing method comprising GSD allocation and gray-value interpolation is proposed to re-project the original adjacent images for eliminating the resolution inconsistency along-track. Finally, the normalized images were stitched based on the perspective transformation with matching corresponding pairs. Experimental results evaluated on the images of Thermal Infrared Spectrometer (TIS) aboard Sustainable Development Goals Satellite-1 (SDGSAT-1) show that the stitching accuracy of the proposed method can be around 0.5 pixels, and more importantly, the ghosting effects and projection distortions are reduced effectively.