Transformer-based language models exhibit In-Context Learning (ICL), where predictions are made adaptively based on context. While prior work links induction heads to ICL through a sudden jump in accuracy, this can only account for ICL when the answer is included within the context. However, an important property of practical ICL in large language models is the ability to meta-learn how to solve tasks from context, rather than just copying answers from context; how such an ability is obtained during training is largely unexplored. In this paper, we experimentally clarify how such meta-learning ability is acquired by analyzing the dynamics of the model’s circuit during training. Specifically, we extend the copy task from previous research into an In-Context Meta Learning setting, where models must infer a task from examples to answer queries. Interestingly, in this setting, we find that there are multiple phases in the process of acquiring such abilities, and that a unique circuit emerges in each phase, contrasting with the single-phases change in induction heads. The emergence of such circuits can be related to several phenomena known in large language models, and our analysis lead to a deeper understanding of the source of the transformer’s ICL ability.

We address the problem of biased gradient estimation in deep Boltzmann machines (DBMs). The existing method to obtain an unbiased estimator uses a maximal coupling based on a Gibbs sampler, but when the state is high-dimensional, it takes a long time to converge. In this study, we propose to use a coupling based on the Metropolis-Hastings (MH) and to initialize the state around a local mode of the target distribution. Because of the propensity of MH to reject proposals, the coupling tends to converge in only one step with a high probability, leading to high efficiency. We find that our method allows DBMs to be trained in an end-to-end fashion without greedy pretraining. We also propose some practical techniques to further improve the performance of DBMs. We empirically demonstrate that our training algorithm enables DBMs to show comparable generative performance to other deep generative models, achieving the FID score of 10.33 for MNIST.

Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variable models owing to costly datapoint-wise sampling iterations and slow convergence. This paper proposes the amortized Langevin dynamics (ALD), wherein datapoint-wise MCMC iterations are entirely replaced with updates of an encoder that maps observations into latent variables. This amortization enables efficient posterior sampling without datapoint-wise iterations. Despite its efficiency, we prove that ALD is valid as an MCMC algorithm, whose Markov chain has the target posterior as a stationary distribution under mild assumptions. Based on the ALD, we also present a new deep latent variable model named the Langevin autoencoder (LAE). Interestingly, the LAE can be implemented by slightly modifying the traditional autoencoder. Using multiple synthetic datasets, we first validate that ALD can properly obtain samples from target posteriors. We also evaluate the LAE on the image generation task, and show that our LAE can outperform existing methods based on variational inference, such as the variational autoencoder, and other MCMC-based methods in terms of the test likelihood.

In industrial assembly tasks, the position of an object grasped by the robot has to be known with high precision in order to insert or place it. In real applications, this problem is commonly solved by jigs that are specially produced for each part. However, they significantly limit flexibility and are prohibitive when the target parts change often, so a flexible method to localize parts with high accuracy after grasping is desired. To solve this problem, we propose a method that can estimate the position of an object in the robot's hand to sub-millimeter precision, and can improve its estimate incrementally, using only minimal calibration and a force sensor. Our method is applicable to any robotic gripper and any rigid object that the gripper can hold, and requires only a force sensor. We demonstrate that the method can determine the position of an object to a precision of under 1 mm without using any part-specific jigs or equipment.

Estimation of joint torque is an important objective in the analyses of human motion. In particular, many applications seek to discern torque during a desired human motion, which is equivalent to solving the inverse dynamics. The computed torque method is a conventional means of calculating inverse dynamics. The obtained torque, however, invariably includes errors resulting from inexact inertial and viscoelastic parameters. This paper presents a method for solving the inverse dynamics of a human arm passively during tracking. To achieve precise human motion tracking, iterative learning control is used for motion generation. Some experiments that target a human arm are executed to validate the proposed method.

Generally, point-to-point control for a completely restrained (CR) parallel-wire-driven system requires a balancing internal force to prevent slackening of wires, along with a feedback term based on some displacement sensor. This paper specifically describes CR systems' internal force properties, then presents the possibility of motion convergence at a desired position when the internal force balancing at a position is given as sensorless feedforward input. Subsequently, we use the property of internal force positively for sensorless position control. This positioning method is applicable for low-cost manipulation, which does not require high accuracy, and for emergency positioning of systems when sensors malfunction.