Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct'' their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating ``error-correction'' data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.

Recent advances in language models have demonstrated their capability to solve mathematical reasoning problems, achieving near-perfect accuracy on grade-school level math benchmarks like GSM8K. In this paper, we formally study how language models solve these problems. We design a series of controlled experiments to address several fundamental questions: (1) Can language models truly develop reasoning skills, or do they simply memorize templates? (2) What is the model's hidden (mental) reasoning process? (3) Do models solve math questions using skills similar to or different from humans? (4) Do models trained on GSM8K-like datasets develop reasoning skills beyond those necessary for solving GSM8K problems? (5) What mental process causes models to make reasoning mistakes? (6) How large or deep must a model be to effectively solve GSM8K-level math questions? Our study uncovers many hidden mechanisms by which language models solve mathematical questions, providing insights that extend beyond current understandings of LLMs.

Virtual reality (VR) technology has shown great potential in mimicking real-world environments for training and collaboration purposes. This study unveils an innovative VR-based collaborative assembly system, uniquely developed on the Unity platform, offering unprecedented interactive capabilities. Unity, a leading VR development toolkit, offers robust 3D rendering and simulation. Using Unity, the system creates an immersive virtual environment where multiple users can collaborate in real-time on assembly tasks using VR headsets. Intuitive virtual tools and interfaces are implemented for grabbing, rotating, and connecting virtual components. The real-time physics engine ensures authentic assembly experiences. Voice chat and gesture interactions are incorporated to facilitate communication and cooperation. Additionally, visual aids and guidance are provided through integrated cameras and artificial intelligence capabilities. The system aims to deliver an immersive and interactive virtual assembly experience. Further studies can focus on evaluating the user experience and task performance.