We present a strategy for te problem of robotic button mushroom harvesting (Agaricus Bisporus) that involves a Real2Sim2Real pipeline with dynamic scene reconstruction and a Model Predictive Path Integral (MPPI) control & planning architecture for generating optimal uprooting motion primitives based on a physics engine simulation framework. Given the complex, non-linear, anisotropic material properties of the mushrooms in combination with the multiple failure-mode modalities involved, we design a simulation framework around the PyBullet rigid-body-physics engine by utilizing first-order approximations of the equivalent continuum mechanics models. By exploiting the computational efficiency of the aforementioned simulation framework, we directly apply the MPPI control framework to generate offline optimal mushroom uprooting motion primitives, defining a set of cost objectives for an optimal and within-constraint harvesting plan. We show that with this planning strategy, the “root-bending” action emerges autonomously for the case of a single mushroom as an optimal uprooting maneuver, which corresponds well to empirical knowledge obtained by expert pickers. A video demonstration of the proposed architecture can be found in https://youtu.be/k38ePBsBego.

Cumulatively developing robots offer a unique opportunity to reenact the constant interplay between neural mechanisms related to learning, memory, prospection, and abstraction from the perspective of an integrated system that acts, learns, remembers, reasons, and makes mistakes. Situated within such interplay lie some of the computationally elusive and fundamental aspects of cognitive behavior: the ability to recall and flexibly exploit diverse experiences of one's past in the context of the present to realize goals, simulate the future, and keep learning further. This article is an adventurous exploration in this direction using a simple engaging scenario of how the humanoid iCub learns to construct the tallest possible stack given an arbitrary set of objects to play with. The learning takes place cumulatively, with the robot interacting with different objects (some previously experienced, some novel) in an open-ended fashion. Since the solution itself depends on what objects are available in the "now," multiple episodes of past experiences have to be remembered and creatively integrated in the context of the present to be successful. Starting from zero, where the robot knows nothing, we explore the computational basis of organization episodic memory in a cumulatively learning humanoid and address (1) how relevant past experiences can be reconstructed based on the present context, (2) how multiple stored episodic memories compete to survive in the neural space and not be forgotten, (3) how remembered past experiences can be combined with explorative actions to learn something new, and (4) how multiple remembered experiences can be recombined to generate novel behaviors (without exploration). Through the resulting behaviors of the robot as it builds, breaks, learns, and remembers, we emphasize that mechanisms of episodic memory are fundamental design features necessary to enable the survival of autonomous robots in a real world where neither everything can be known nor can everything be experienced.

We all `inhabit' continuously changing unstructured worlds where neither everything can be known nor can everything be experienced. The lifelong interplay between neural mechanisms associated with learning, memory, prospection and abstraction play a fundamental role in enabling cognitive agents to effortlessly connect their `past' with the `available present' and `possible future', most often in the context of realization of one's internal goals (and taking into account neurobiological and physical constraints related to one's embodiment). Investigation into the computational and neural basis underlying such mechanisms have value both from an intrinsic viewpoint of better understanding our own selves and at the same time creating artifacts that can flexibly assist us in the environments we inhabit and create (domestic, industrial, several others). Simply put, beyond a certain point a software programmer cannot travel the journey of a cognitive robot, they must learn to do it themselves! Enabling them to do so offers us with a unique opportunity to reenact the gradual process of infant developmental learning and investigate deeper into the underlying interplay between multiple fundamental cognitive processes from the perspective of an “integrated system” (human or humanoid that perceives, acts, learns, forgets and reasons). The article reports some advanced experiments conducted on the baby humanoid iCub to this effect, mainly 1. Learning to push and begin to anticipate how objects move, use this knowledge and 2. Learning to build the tallest possible stack given a random set of objects to play with (learning going on cumulatively in an open ended fashion). These playful experiments are used to summarize the gradually evolving cognitive architecture of the young EU funded project DARWIN.

The subjective ease with which we move gracefully in constraint filled uncertain environments often masks the enormously complex integrative apparatus needed to spell synergy among the thousands of sensors, joints, musculo-skeletal units and neuronal populations that contribute to any act's planning and execution. In this paper, we apply the computational framework of passive motion paradigm for task specific composition and coordination of the movements of a limb, network of limbs (e.g. left arm-waist-right arm) or networks of external objects coupled to the body of the 53 degrees of freedom humanoid robot `iCub'. The basic PMP model is further extended by formulation of a pair of branching nodes that allow compositionality and transfer of force fields from one relaxation network to another. The generality of the proposed approach is further illustrated using simulations of whole body reaching (WBR) tasks from a quiet standing posture that recruits virtually all the joints of the upper limbs, lower limbs, and trunk, binding together a large number of degrees of freedom into a functional unit that combines a focal task (reaching a target with the hand) and a postural task (keeping the projection of the center of mass within the bipedal support area). Preliminary comparisons of the solutions generated by the computational model with the movements of human subjects performing similar WBR tasks are presented.

Unlike a purely reactive system where the motor output is exclusively controlled by the actual sensory input, a cognitive system must be capable of running mental processes which virtually simulate action sequences aimed at achieving a goal. The mental process either attempts to find a feasible course of action compatible with a number of constraints (Internal, Environmental, Task Specific etc) or selects it from a repertoire of previously learned actions, according to the parameters of the task. If neither reasoning process succeeds, a typical backup strategy is to look for a tool that might allow the operator to match all the task constraints. This further necessitates having the capability to alter ones own goal structures to generate sub-goals which must be successfully accomplished in order to achieve the primary goal. In this paper, we introduce a forward/inverse motor control architecture (FMC/IMC) that relaxes an internal model of the overall kinematic chain to a virtual force field applied to the end effector, in the intended direction of movement. This is analogous to the mechanism of coordinating the motion of a wooden marionette by means of attached strings. The relaxation of the FMC/IMC pair provides a general solution for mentally simulating an action of reaching a target position taking into consideration a range of geometric constraints (range of motion in the joint space, internal and external constraints in the workspace) as well as effort-related constraints (range of torque of the actuators, etc.). In case, the forward simulation is successful, the movement is executed; otherwise the residual "error" or measure of inconsistency is taken as a starting point for breaking the action plan into a sequence of sub actions. This process is achieved using a recurrent neural network (RNN) which coordinates the overall reasoning process of framing and issuing goals to the forward inverse models, searching for alternatives tools in solution space and formation of sub-goals based on past context knowledge and present inputs. The RNN + FMC/IMC system is able to successfully reason and coordinate a diverse range of reaching and grasping sequences with/without tools. Using a simple robotic platform (5 DOF Scorbot arm + Stereo vision) we present results of reasoning and coordination of arm/tool movements (real and mental simulation) specifically directed towards solving the classical 2-stick paradigm from animal reasoning at a non linguistic level.