In this paper, we present a benchmark to pressure-test today’s frontier models’ multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1 as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We study increasing amounts of expert demonstrations in the context — from no demonstrations to 512 full episodes. Across our tasks, models rarely manage to fully reach expert performance, and often, presenting more demonstrations has little effect. Some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observations as text or images and the impact of chain-of-thought prompting. To help quantify the impact of other approaches and future innovations, we open source our benchmark that covers the zero-, few-, and many-shot regimes in a unified evaluation.

Recent work in prosthetic devices suggests that forearm surface electromyography (sEMG) is a promising technology for human-computer interactions. Specifically, a system able to detect individual finger movement can have many clinical and nonclinical applications. Popular consumer-grade sEMG armbands are limited by their fixed electrode arrangement, which can negatively affect the classification of subtle finger gestures. We propose a low-cost, 3D-printed armband with fully adjustable electrode placement for the detection of single-finger tapping motions. We trained machine learning classifiers on features extracted from eight-channel sEMG signals to detect movement from nine fingers. We obtained a classification accuracy of 71.5 ± 1.1% for a K-Nearest Neighbours (KNN) classifier using features extracted from 500 ms windows of sEMG data. Moreover, a KNN model trained on 200 ms windows from a subset of particularly clean data obtained an accuracy of 93.0 ± 0.5%. We also introduce a novel haptic feedback mechanism to improve user experience when using the armband, and propose an augmented reality typing interface as a potential application of our armband.