Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting pointbased affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.

In this paper, we present PortaChrome, a portable light source that can be attached to everyday objects to reprogram the color and texture of surfaces that come in contact with them. When PortaChrome makes contact with objects previously coated with photochromic dye, the UV and RGB LEDs inside PortaChrome create multi-color textures on the objects. In contrast to prior work, which used projectors for the color-change, PortaChrome has a thin and flexible form factor, which allows the color-change process to be integrated into everyday user interaction. Because of the close distance between the light source and the photochromic object, PortaChrome creates color textures in less than 4 minutes on average, which is 8 times faster than prior work. We demonstrate PortaChrome with four application examples, including data visualizations on textiles and dynamic designs on wearables.