Generative AI systems optimized for efficient artifact production can inadvertently bypass the cognitive labor that gives knowledge work its value. Through a Research through Design study in corporate goal-setting—a reflective practice increasingly hollowed by bureaucratic pressures—we propose and evaluate a design principle: cognitive outcomes first, material artifacts second. We operationalize this through a prototype using cognitive scaffolds that require users to articulate contextual nuances, constraints, and strategic realities before generating artifacts. Evaluation with 16 employees revealed that 75% recognized persistent questioning as productive rigor yielding “cognitive delta” (new insight). A five-month follow-up confirmed durability of cognitive outcomes beyond the study context. We contribute: (1) Cognitive outcomes first, a principle establishing cognitive labor as the primary product and artifacts as proof of work; (2) Instrumental drift, an analytic lens revealing how efficiency-optimized AI decouples instruments from purposes; and (3) empirical demonstration that persistent inquiry succeeds when designed for cognitive engagement, with users recognizing epistemic value despite interaction costs.

Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information. This work directly classifies mouthing instances to their corresponding words in the spoken language while exploring the potential of transfer learning from Visual Speech Recognition (VSR) to mouthing recognition in German Sign Language. We leverage three VSR datasets: one in English, one in German with unrelated words and one in German containing the same target words as the mouthing dataset, to investigate the impact of task similarity in this setting. Our results demonstrate that multi-task learning improves both mouthing recognition and VSR accuracy as well as model robustness, suggesting that mouthing recognition should be treated as a distinct but related task to VSR. This research contributes to the field of SLR by proposing knowledge transfer from VSR to SLR datasets with limited mouthing annotations.

We present the acquisition process and the data of DGS-Fabeln-1, a parallel corpus of German text and videos containing German fairy tales interpreted into the German Sign Language (DGS) by a native DGS signer. The corpus contains 573 segments of videos with a total duration of 1 hour and 32 minutes, corresponding with 1428 written sentences. It is the first corpus of semi-naturally expressed DGS that has been filmed from 7 angles, and one of the few sign language (SL) corpora globally which have been filmed from more than 3 angles and where the listener has been simultaneously filmed. The corpus aims at aiding research at SL linguistics, SL machine translation and affective computing, and is freely available for research purposes at the following address: https://doi.org/10.5281/zenodo.10822097.

State-of-the-art techniques common to low resource Machine Translation (MT) are applied to improve MT of spoken language text to Sign Language (SL) glosses.In our experiments, we improve the performance of the transformer-based models via (1) data augmentation, (2) semi-supervised Neural Machine Translation (NMT), (3) transfer learning and (4) multilingual NMT.The proposed methods are implemented progressively on two German SL corpora containing gloss annotations.Multilingual NMT combined with data augmentation appear to be the most successful setting, yielding statistically significant improvements as measured by three automatic metrics (up to over 6 points BLEU), and confirmed via human evaluation.Our best setting outperforms all previous work that report on the same test-set and is also confirmed on a corpus of the American Sign Language (ASL).

This paper is a brief summary of the First WMT Shared Task on Sign Language Translation (WMT-SLT22), a project partly funded by EAMT. The focus of this shared task is automatic translation between signed and spoken languages. Details can be found on our website (https://www.wmt-slt.com/) or in the findings paper (Müller et al., 2022).

Despite the importance of mouth actions in Sign Languages, previous work on Automatic Sign Language Recognition (ASLR) has limited use of the mouth area.Disambiguation of homonyms is one of the functions of mouth actions, making them essential for tasks involving ambiguous hand signs.To measure their importance for ASLR, we trained a classifier to recognize ambiguous hand signs.We compared three models which use the upper body/hands area, the mouth, and both combined as input.We found that the addition of the mouth area in the model resulted in the best accuracy, giving an improvement of 7.2% and 4.7% on the validation and test set, while allowing disambiguation of the hand signs for most of the cases.In cases where the disambiguation failed, it was observed that the signers in the video samples occasionally didn't perform mouthings.In a few cases, the mouthing was enough to achieve full disambiguation of the signs.We conclude that further investigation on the modelling of the mouth region can be beneficial of future ASLR systems.

We investigate whether using semi-supervised learning (SSL) methods can be beneficial for the task of word-level Quality Estimation of Machine Translation in low resource conditions. We show that the Mean Teacher network can provide equal or significantly better MCC scores (up to +12%) than supervised methods when a limited amount of labeled data is available. Additionally, following previous work on SSL, we investigate Pseudo-Labeling in combination with SSL, which nevertheless does not provide consistent improvements.

This paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried out with regular expressions. We describe the creation and implementation of the test suite in detail, providing a full list of all categories and phenomena. Furthermore, we present various exemplary applications of our test suite that have been implemented in the past years, like contributions to the Conference of Machine Translation, the usage of the test suite and MT outputs for quality estimation, and the expansion of the test suite to the language pair Portuguese–English. We describe how we tracked the development of the performance of various systems MT systems over the years with the help of the test suite and which categories and phenomena are prone to resulting in MT errors. For the first time, we also make a large part of our test suite publicly available to the research community.

In this paper, we describe the current main approaches to sign language translation which use deep neural networks with videos as input and text as output. We highlight that, under our point of view, their main weakness is the lack of generalization in daily life contexts. Our goal is to build a state-of-the-art system for the automatic interpretation of sign language in unpredictable video framing conditions. Our main contribution is the shift from image features to landmark positions in order to diminish the size of the input data and facilitate the combination of data augmentation techniques for landmarks. We describe the set of hypotheses to build such a system and the list of experiments that will lead us to their verification.

This paper describes the proof-of-concept evaluation for a system that provides translation of speech to virtually performed sign language on augmented reality (AR) glasses. The discovery phase via interviews confirmed the idea for a signing avatar displayed within the users field of vision through AR glasses. In the evaluation of the first prototype through a wizard-of-Oz-experiment, the presented AR solution received a high acceptance rate among deaf and hard-of-hearing persons. However, the machine learning based method used to generate sign language from video still lacks the required accuracy for fully preserving comprehensibility. Signed sentences with large recognisable arm movements were understood better than sentences relying mainly on finger movements, where only a small interaction space is visible.

Stochastic differential equations (SDEs) are widely used to model systems affected by random processes. In general, the analysis of an SDE model requires numerical solutions to be generated many times over multiple parameter combinations. However, this process often requires considerable computational resources to be practicable. Due to the embarrassingly parallel nature of the task, devices such as multi-core processors and graphics processing units (GPUs) can be employed for acceleration. Here, we present SODECL (https://github.com/avramidis/sodecl), a software library that utilizes such devices to calculate multiple orbits of an SDE model. To evaluate the acceleration provided by SODECL, we compared the time required to calculate multiple orbits of an exemplar stochastic model when one CPU core is used, to the time required when using all CPU cores or a GPU. In addition, to assess scalability, we investigated how model size affected execution time on different parallel compute devices. Our results show that when using all 32 CPU cores of a high-end high-performance computing node, the task is accelerated by a factor of up to ≈6.7, compared to when using a single CPU core. Executing the task on a high-end GPU yielded accelerations of up to ≈4.5, compared to a single CPU core.