A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner.We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism.GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training.It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss.We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion.We pre-train GPST on OpenWebText, a corpus with 9 billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation.Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.1 * Equal contribution, see appendix A.7 for details.

We present ReCAT, a recursive composition augmented Transformer that is able to explicitly model hierarchical syntactic structures of raw texts without relying on gold trees during both learning and inference. Existing research along this line restricts data to follow a hierarchical tree structure and thus lacks inter-span communications. To overcome the problem, we propose a novel contextual inside-outside (CIO) layer that learns contextualized representations of spans through bottom-up and top-down passes, where a bottom-up pass forms representations of high-level spans by composing low-level spans, while a top-down pass combines information inside and outside a span. By stacking several CIO layers between the embedding layer and the attention layers in Transformer, the ReCAT model can perform both deep intra-span and deep inter-span interactions, and thus generate multi-grained representations fully contextualized with other spans. Moreover, the CIO layers can be jointly pre-trained with Transformers, making ReCAT enjoy scaling ability, strong performance, and interpretability at the same time. We conduct experiments on various sentence-level and span-level tasks. Evaluation results indicate that ReCAT can significantly outperform vanilla Transformer models on all span-level tasks and recursive models on natural language inference tasks. More interestingly, the hierarchical structures induced by ReCAT exhibit strong consistency with human-annotated syntactic trees, indicating good interpretability brought by the CIO layers.

SolarSense is a self-powered sensing system for gesture recognition using solar cell arrays, thereby offering a sustainable approach to human-computer interaction (HCI) within industrial settings. The system effectively employed the sensing and energy harvesting capabilities of solar cells, achieving over 97.0% accuracy in recognizing diverse gestures. The design incorporates a low-power wireless data acquisition chip, a signal processing framework, and a user interface to realize robotic control and text input applications. SolarSense enhances HCI with its eco-friendly and user-centric approach, which is suitable for Internet of things (IoT) scenarios. Demo: https://youtu.be/RmPolChw_c4.

The energy-efficient train control (EETC) problem is investigated in this article. And a soft actor-critic (SAC)-based method is proposed to optimize the train driving strategy. First, EETC problem is converted to the inverse problem, i.e., minimizing the trip time of the journey with constant energy consumption. Based on the conversion, the EETC problem is reformulated as a finite Markov decision process, which can be solved by deep reinforcement learning algorithms. Second, an optimization method based on the SAC method is designed to calculate the optimal driving strategy of the train with introducing the reservoir sampling method. Finally, some case studies are conducted to verify the effectiveness and performance of the proposed method. Simulation results demonstrate that a good energy-saving performance can be achieved. In single interval, the SAC-based method can reduce about 1.65% of the energy consumption compared with numerical method. And the energy consumption reduction can be extended to be 6.49% when the proposed approach is applied in multiple intervals.