Large Language Models (LLMs) are rapidly saturating existing benchmarks, necessitating new open-ended evaluations. We introduce the Factorio Learning Environment (FLE), based on the game of Factorio, that tests agents in long-term planning, spatial reasoning, program synthesis, and resource optimization. FLE provides exponentially scaling challenges -- from basic automation to complex factories processing millions of resource units per second. We provide two settings: (1) open-play with the open-ended task of building the largest factory on an procedurally generated map and (2) lab-play consisting of 33 bounded tasks accross three settings with fixed resources. We demonstrate across both settings that models still lack strong spatial reasoning. In lab-play, we find that LLMs exhibit promising short-horizon skills, yet are unable to operate effectively in constrained environments, reflecting limitations in error analysis. In open-play, while LLMs discover automation strategies that improve growth (e.g electric-powered drilling), they fail to achieve complex automation (e.g electronic-circuit manufacturing)

Detecting emotions and affect in text has received enormous attention in recent years, and yet majority of the works in this area reduce the nuanced emotional responses into ‘positive’, ‘negative’ and ‘neutral’. In this paper, we introduce a novel multi-dimensional affect in poetry (POCA) dataset for sentiment analysis annotated using the Geneva Emotion Wheel (GEW), to capture and analyse the multi-dimensional affect evoked in listeners. The POCA dataset is based on poems and their corresponding recitals from an online poetry database where recitals are curated by the website, and performed by the poet or an approved artist. The POCA dataset contains 330 poems (text and audio), from the English language, each of which is annotated across 20 different emotion classes, by 5 listeners. A subset of the dataset (50 poems) have also been annotated by an inlab study by 3 listeners each while their Electrodermal activity (EDA) was being recorded. As a proof of concept, we (i) introduce representative problem formulations to be addressed by machine learning approaches using the POCA dataset, from single emotion recognition (e.g., does this poem evoke joy?) to continuous affect prediction (e.g., what level arousal and valence does this poem evoke?), (ii) provide baseline results for text-based affect recognition using several classification and regression models, and (iii) provide baseline results for EDA-based affect prediction. Our results show that (i) for text-based affect recognition, classical approaches can provide as accurate results as their fine-tuned neural network counterparts, and (ii) in EDA-based affect prediction, in general there is a strong relation between the EDA signals and the self-reported valence and arousal quadrants, while predictions are better for arousal than valence.

We propose two novel methodologies for the automatic generation of rhythmic poetry in a variety of forms.The first approach uses a neural language model trained on a phonetic encoding to learn an implicit representation of both the form and content of English poetry.This model can effectively learn common poetic devices such as rhyme, rhythm and alliteration.The second approach considers poetry generation as a constraint satisfaction problem where a generative neural language model is tasked with learning a representation of content, and a discriminative weighted finite state machine constrains it on the basis of form.By manipulating the constraints of the latter model, we can generate coherent poetry with arbitrary forms and themes.A large-scale extrinsic evaluation demonstrated that participants consider machine-generated poems to be written by humans 54% of the time.In addition, participants rated a machinegenerated poem to be the most human-like amongst all evaluated.