Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.

The automatic control of mobile devices is essential for efficiently performing complex tasks that involve multiple sequential steps.However, these tasks pose significant challenges due to the limited environmental information available at each step, primarily through visual observations.As a result, current approaches, which typically rely on reactive policies, focus solely on immediate observations and often lead to suboptimal decision-making.To address this problem, we propose Foresighted Planning with World Model-Driven Code Execution (FPWC), a framework that prioritizes natural language understanding and structured reasoning to enhance the agent's global understanding of the environment by developing a task-oriented, refinable world model at the outset of the task.Foresighted actions are subsequently generated through iterative planning within this world model, executed in the form of executable code.Extensive experiments conducted in simulated environments and on real mobile devices demonstrate that our method outperforms previous approaches, particularly achieving a 44.4% relative improvement in task success rate compared to the state-of-the-art in the simulated environment.