Since inevitable distribution shifts are encountered during test time in practice, test-time adaptation (TTA) presents a promising solution by recalibrating the model online using only an unlabeled test data stream. However, TTA often suffers from issues such as catastrophic forgetting caused by continuously changing environments, as it relies on self-training. Contemporary solutions attempt to mitigate this by anchoring TTA to a static source model, such as stochastic parameter restoration or periodic parameter reset, which restrict model flexibility. Moreover, different layers may exhibit varying sensitivities to distribution shifts, sometimes even showing opposite shift trends, yet prior methods treat all layers homogeneously. Motivated by this, we propose a layer-wise parameter robustness method that autonomously identifies important parameters in different layers for selective weighting by measuring the sharpness of parameter surface. Further in-depth experiments on various benchmarks demonstrate the robustness and effectiveness of our proposed method. Our code is available at https://github.com/ioslide/prda_tta.

We focus on the transportation-aware trajectory recovery problem, which is distinct from the conventional vehicle-based trajectory recovery, facing three major challenges: heterogeneity, personalization and efficiency. For the heterogeneity, the velocity of the mobile object is intrinsically correlated with the specific transportation mode, containing inherent heterogeneity. For the personalization, the trajectory data is complicated by substantial variations in users, which are different in personalized behaviors. For the efficiency, previous works mostly employ sequence-to-sequence framework which limits their efficiency due to the auto-regressive inference pattern. To address these challenges, we design a novel efficient and effective multi-modal deep model, coined as PTrajRec, for transportation-aware trajectory recovery. Specifically, we initially embed location, behavior, and transportation mode modalities in distinct channels, which not only reflect spatio-temporal information encapsulated in location sequences but also introduce the heterogeneity and personalization characteristics associated with mode and behavior sequences. For further modeling these modalities, we employ the auto-correlation mechanism to learn periodic dependencies on the temporal dimension and the graph attention mechanism to learn road network dependencies on the spatial dimension. At last, we propose a dual-view constraint mechanism to assist the efficient trajectory recovery framework and design three auxiliary tasks to address the inherent heterogeneity and efficiency design. Extensive experimental results on two real-world datasets demonstrate the superiority of our proposed method compared to state-of-the-art baselines with reduced computation cost and excellent performance.

Test-time adaptation (TTA) aims to adapt the neural network to the distribution of the target domain using only unlabeled test data. Most previous TTA methods have achieved success under mild conditions, such as considering only a single or multiple independent static domains. However, in real-world settings, the test data is sampled in a correlated manner and the test environments undergo continual changes over time, which may cause previous TTA methods to fail in practical noise scenarios, i.e., independent noise distribution shifts, continual noise distribution shifts, and continual mixed distribution shifts. To address these issues, we elaborate a Stable Test-time Adaptation Framework, called STAF, to stabilize the adaptation process. Specifically, to boost model robustness to noise distribution shifts, we present a multi-stream perturbation consistency method, enabling weak-to-strong views to be consistent, guided by the weak view from the original sample. Meanwhile, we develop a reliable memory-based corrector which utilizes reliable snapshots between the anchor model and the adapt model to correct prediction bias. Furthermore, we propose a dynamic parameter restoration strategy to alleviate error accumulation and catastrophic forgetting that takes into account both the distribution shift and sample adaptation degree. Extensive experiments demonstrate the robustness and effectiveness of STAF, which pushes the boundaries of test-time adaptation to more realistic scenarios and paves the way for stable deployment of real-world applications.

Test-Time Adaptation (TTA) aims to provide a deployed agent capable of adapting to the target domain distribution using only unlabeled test data. Most existing TTA methods have achieved success under mild conditions, such as independently sampled data from a single or multiple static domains. However, these attempts may cause the deployed agent to fail in dynamic scenarios, where the test environment undergoes continuous changes over time. Previous TTA methods attempt to improve the quality of pseudo-labels by designing data augmentation strategies based on domain information. However, this violates the TTA paradigm, i.e., no characteristics of the dataset can be known in advance. Motivated by this, we propose a Robust Dual-stream Perturbation Test-time adaptation approach, called RDPT, to further stabilize the deployed agent from two aspects: (1) boost the robustness to noisy samples by encouraging the model to be consistent with its original image-level perturbation samples; (2) dynamically filtering samples with high entropy values, taking into account the discrepancy between the original and perturbation samples. Extensive experiments on the CIFAR10C and CIFAR100C benchmarks demonstrate the effectiveness of our method in dynamic scenarios, which is also easy to implement and deploy.1

While there has been remarkable progress recently in the fields of manipulation and locomotion, mobile manip-ulation remains a long-standing challenge. Compared to locomotion or static manipulation, a mobile system must make a diverse range of long-horizon tasks feasible in un-structured and dynamic environments. While the applications are broad and interesting, there are a plethora of chal-lenges in developing these systems such as coordination be-tween the base and arm, reliance on onboard perception for perceiving and interacting with the environment, and most importantly, simultaneously integrating all these parts to-gether. Prior works approach the problem using disentangled modular skills for mobility and manipulation that are trivially tied together. This causes several limitations such as compounding errors, delays in decision-making, and no whole-body coordination. In this work, we present a re-active mobile manipulation framework that uses an active visual system to consciously perceive and react to its en-vironment. Similar to how humans leverage whole-body and hand-eye coordination, we develop a mobile manipu-lator that exploits its ability to move and see, more specifically - to move in order to see and to see in order to move. This allows it to not only move around and interact with its environment but also, choose “when” to perceive “what” using an active visual system. We observe that such an agent learns to navigate around complex cluttered sce-narios while displaying agile whole-body coordination using only ego-vision without needing to create environment maps. Videos are available at https://spin-robot.github.io

Travel time estimation(TTE) is a critical component of intelligent transportation systems. To achieve efficient and accurate trajectory-based travel time estimation, it is essential to design a streamlined model that reduces computation and memory costs. However, this is challenging as traditional deep neural networks are limited in their calculation capabilities and can be cumbersome due to the high number of model parameters. To overcome these challenges, we propose a novel approach to travel time estimation, utilizing a well-designed deep neural network model called Knowledge Distillation for Travel Time Estimation (KDTTE). By implementing knowledge distillation techniques, the model’s computational and memory requirements are reduced, while simultaneously improving its accuracy. The student model leverages the knowledge of the Teacher model to learn features it would not have been able to on its own, thereby enhancing the overall accuracy of the model. Our approach, referred to as KDTTE, was tested on two real-world datasets and showed improved accuracy compared to nine state-of-the-art baselines, demonstrating a 34.0% and 86.8% increase in accuracy on the Chengdu and Porto datasets, respectively.

Learning from visual data opens the potential to accrue a large range of manipulation behaviors by leveraging human demonstrations without specifying each of them mathe-matically, but rather through natural task specification. In this paper, we present Learning by Watching (LbW), an algorithmic framework for policy learning through imitation from a single video specifying the task. The key insights of our method are two-fold. First, since the human arms may not have the same morphology as robot arms, our framework learns unsupervised human to robot translation to overcome the morphology mis-match issue. Second, to capture the details in salient regions that are crucial for learning state representations, our model performs unsupervised keypoint detection on the translated robot videos. The detected keypoints form a structured representation that contains semantically meaningful information and can be used directly for computing reward and policy learning. We evaluate the effectiveness of our LbW framework on five robot manipulation tasks, including reaching, pushing, sliding, coffee making, and drawer closing. Extensive experimental evaluations demonstrate that our method performs favorably against the state-of-the-art approaches. More results and analysis are available at pair.toronto.edu/lbw-kp/.