Human-centric motion control in video generation remains a critical challenge, particularly when jointly controlling camera movements and human poses in scenarios like the iconic Grammy Glambot moment. While recent video diffusion models have made significant progress, existing approaches struggle with limited motion representations and inadequate integration of camera and human motion controls. In this work, we present TokenMotion, the first DiT-based video diffusion framework that enables fine-grained control over camera motion, human motion, and their joint interaction. We represent camera trajectories and human poses as spatio-temporal tokens to enable local control granularity. Our approach introduces a unified modeling framework utilizing a decouple-and-fuse strategy, bridged by a human-aware dynamic mask that effectively handles the spatially-and-temporally varying nature of combined motion signals. Through extensive experiments, we demonstrate TokenMotion’s effectiveness across both text-to-video and image-to-video paradigms, consistently outperforming current state-of-the-art methods in human-centric motion control tasks. Our work represents a significant advancement in controllable video generation, with particular relevance for creative production applications.

The exploration of mutual-benefit cross-domains has shown great potential toward accurate self-supervised depth estimation. In this work, we revisit feature fusion between depth and semantic information and propose an efficient local adaptive attention method for geometric aware representation enhancement. Instead of building global connections or deforming attention across the feature space without restraint, we bound the spatial interaction within a learnable region of interest. In particular, we leverage geometric cues from semantic information to learn local adaptive bounding boxes to guide unsupervised feature aggregation. The local areas preclude most irrelevant reference points from attention space, yielding more selective feature learning and faster convergence. We naturally extend the paradigm into a multi-head and hierarchic way to enable the information distillation in different semantic levels and improve the feature discriminative ability for fine-grained depth estimation. Extensive experiments on the KITTI dataset show that our proposed method establishes a new state-of-the-art in self-supervised monocular depth estimation task, demonstrating the effectiveness of our approach over former Transformer variants.

This paper presents a multistage, real-time optimization algorithm to perform visual searches. A cost function relates the success of the visual search at any given pose in the workspace. This function is measurable at every camera pose, but it is not necessarily known as a function of changing pose, and its gradient with respect to the workspace or joint space coordinates is not known. In the first stage, a camera is constrained on a fixed sphere centered on the robot and searches for a general target direction by multidirectional direct search. In the second stage, the camera is constrained on a series of spheres centered on refined estimates of the search target's position. A series of extremum seeking control (ESC) algorithms are executed to search the best pose to observe the target. Since there is no existing method to detect when ESC has converged, we propose a novel frequency domain criteria to decide when to terminate the current ESC and launch another on the next candidate sphere. We prove that, under certain conditions, ESC will converge to known neighborhood of the optimal point in finite time. Simulations and experiments are presented to illustrate the effectiveness of the proposed algorithm.

In this paper, we present a novel camera-projector system for assisting robot-human interaction. The system is comprised of a stereo camera pair and a DLP projector. The proposed system provides feedback information indicating the robot's perception of the environment and what action a human user desires. Feedback is delivered by iteratively spotlighting objects in the environment using the projector. When the spotlight lands on an object that the human user wants the robot to retrieve, he or she can confirm the object selection, and the robot will perform a grasping task to retrieve the selected object. In this investigation, the proposed camera-projector performs three tasks: 1) Actively detect visually salient objects in the scene from the two camera views using a visual attention model. 2) Locate the detected objects using a trifocal tensor based object matching method, and project a spot of light on the objects to provide information back to the human user. 3) Reconstruct the 3D position of the target to plan robot motion and conduct vision-based control to move a robot manipulator to grasp the object. Experimental results of human-directed object grasping task are presented to demonstrate the functions of the proposed system.

This paper presents a novel navigation and control system for autonomous mobile robots that includes path planning, localization, and control. A unique vision-based pose and velocity estimation scheme utilizing both the continuous and discrete forms of the Euclidean homography matrix is fused with inertial and optical encoder measurements to estimate the pose, orientation, and velocity of the robot and ensure accurate localization and control signals. A depth estimation system is integrated in order to overcome the loss of scale inherent in vision-based estimation. A path following control system is introduced that is capable of guiding the robot along a designated curve. Stability analysis is provided for the control system and experimental results are presented that prove the combined localization and control system performs with high accuracy.

This paper presents a novel approach to increase the amount of visual stimuli in sensor measurements using saliency maps. A saliency map is a combination of normalized feature maps in different channels (i.e. color, intensity) to represent the relative strength of visual stimuli in an image. The total saliency is higher when the camera is looking at a scene with more interesting things in the field of view and vise versa. We employ methods of extremum seeking control to find a camera position that corresponds to local maximum saliency value. We combine the global properties of simplex optimization methods with the local search properties and dynamic response of extremum seeking control to create a novel algorithm that is more likely to find a global maximum than conventional extremum seeking control. Simulations and experiments are presented to show the strength of this approach.