Recognizing the physical states of objects and their transformations within videos is crucial for structured video understanding and enabling robust real-world applications, such as robotic manipulation. However, pretrained vision-language models often struggle to capture these nuanced dynamics and their temporal context, and specialized object state recognition frameworks may not generalize to unseen actions or objects. We introduce SAGE (State-Action Graph Embeddings), a novel framework that offers a unified model of physical state transitions by decomposing states into fine-grained, language-described visual concepts that are sharable across different objects and actions. SAGE initially leverages Large Language Models to construct a State-Action Graph, which is then multimodally refined using Vision-Language Models. Extensive experiments show that our method significantly outperforms baselines, generalizes effectively to unseen objects and actions in open-world settings. SAGE improves the prior state-of-the-art by as much as 14.6% on novel state recognition with less than 5% of its inference time.

Auxiliary loss is additional loss besides the main branch loss to help optimize the learning process of neural networks. In order to calculate the auxiliary loss between the feature maps of intermediate layers and the ground truth in the field of semantic segmentation, the size of each feature map must match the ground truth. In all studies using the auxiliary losses with the segmentation models, from what we have investigated, they either use a down-sampling function to reduce the size of the ground truth or use an up-sampling function to increase the size of the feature map in order to match the resolution between the feature map and the ground truth. However, in the process of selecting representative values through down-sampling and up-sampling, information loss is inevitable. In this paper, we introduce Class Probability Preserving (CPP) pooling to alleviate information loss in down-sampling the ground truth in semantic segmentation tasks. We demonstrated the superiority of the proposed method on Cityscapes, Pascal VOC, Pascal Context, and NYU-Depth-v2 datasets by using CPP pooling with auxiliary losses based on seven popular segmentation models. In addition, we propose See-Through Network (SeeThroughNet) that adopts an improved multi-scale attention-coupled decoder structure to maximize the effect of CPP pooling. SeeThroughNet shows cutting-edge results in the field of semantic understanding of urban street scenes, which ranked #1 on the Cityscapes benchmark.

End-to-end learning in deep reinforcement learning based on raw visual input has shown great promise in various tasks involving sensorimotor control. However, complex tasks such as tool use require recognition of affordance and a series of non-trivial subtasks such as reaching the tool, grasping the tool, and wielding the tool. In such tasks, end-to-end approaches with only the raw input (e.g. pixel-wise images) may fail to learn to perform the task or may take too long to converge. In this paper, inspired by the biological sensorimotor system, we explore the use of proprioceptive/kinesthetic inputs (internal inputs for body position and motion) as well as raw visual inputs (exteroception, external perception) for use in affordance learning for tool use tasks. We set up a reaching task in a simulated physics environment (MuJoCo), where the agent has to pick up a T-shaped tool to reach and drag a target object to a designated region in the environment. We used an Actor-Critic-based reinforcement learning algorithm called ACKTR (Actor-Critic using Kronecker-Factored Trust Region) and trained it using various input conditions to assess the utility of proprioceptive/kinesthetic inputs. Our results show that the inclusion of proprioceptive/kinesthetic inputs (position and velocity of the limb) greatly enhances the performance of the agent: higher success rate, and faster convergence to the solution. The lesson we learned is the important factor of the intertwined relationship of exteroceptive and proprioceptive in sensorimotor learning and that although end-to-end learning based on raw input may be appealing, separating the exteroceptive and proprioceptive/kinesthetic factors in the input to the learner, and providing the necessary internal inputs can lead to faster, more effective learning.

Recurrent neural networks (RNNs) often show very complicated temporal behavior. In this paper, we investigate the dynamics of a simple recurrent neural network used in a nontrivial articulated limb control task in a tool use domain. The RNN for the task is evolved by the NeuroEvolution of Augmenting Topologies (NEAT) algorithm. As a non-linear dynamical system, RNNs exhibit strong correlation between their external behavior and internal dynamics. Discovery of the fixed points and limit cycles in the system dynamics will help us understand the roles of neurons and connections in the neural network, and provide insights on how to analyze the function of biology-based nervous systems. We hope our results can provide new criteria to the evaluation of the evolved RNNs, and help the diagnosis of the failures in control problems using neuroevolution.

Tool use requires high levels of cognitive function and is only observed in higher mammals and some avian species such as corvids. In this paper, we will investigate how the capability to use tools can spontaneously emerge in a simulated evolution of a two degree-of-freedom articulated limb. The controller for the limb was evolved as neural circuits that can gradually take on arbitrary topologies (NeuroEvolution of Augmenting Topologies, or NEAT). First, we show how very broad fitness criteria such as distance to the target, number of steps to reach the target, and tool pick-up frequency are enough to give rise to tool using behavior. Second, we analyze the evolved neural circuits to find properties that enable tool use. We expect our results to help us understand the origin of tool use and the kind of neural circuits that enabled such a powerful trait.