With the rapid advancements in machine translation (MT), large language models (LLMs) have demonstrated the ability to generate high-quality translations with relatively low computational resources compared to training neural machine translation (NMT) systems from scratch. Recent studies have shown significant improvements in translation quality by lever- aging in-context learning (ICL), which provides LLMs with context during the translation process. However, when it comes to literary texts, which are inherently informative in content, diverse in genres and complex in sentence structures, current LLMs and existing methodologies struggle to perform effectively. A key factor contributing to this suboptimal performance is the insufficient infusion of classical literary translation knowledge during the pre-training phase of LLMs, which hampers their effectiveness in literary translation tasks. In this paper, we propose a novel method based on syntactic hierarchical Retrieval Augmented Generation (RAG) to address the unique challenges of literary machine translation (LMT) for English-to-Chinese novel translations. Grounded in the theory of elements in fiction, we construct a comprehensive dataset that encompasses four distinct content types commonly found in literary texts. Our dataset consists of manually collected high-quality bilingual parallel translation pairs extracted from four novels, which are utilized for database construction, example retrieval, LLMs fine- tuning and testing. By establishing a maintainable database and designing a hierarchical retrieval system, our method selects semantically relevant examples from the source language to facilitate augmented generation by the LLMs. Through extensive experimentation, we demonstrate the effectiveness of our method in enhancing the translation quality of LLMs. The results indicate significant improvements, with our hierarchical RAG framework yielding an average increase of 25.75% in METEOR scores compared to LLaMA3-70B, 21.31% compared to Google Translate, 11.46% compared to GPT-3.5 and an improvement of more than 10% in GPT-4o scores over all metrics. These empirical findings substantiate the superiority of our approach, establishing its potential to advance the field of literary machine translation.

Task-oriented grasping (TOG) involves grasping specific parts of an object based on a task instruction. Existing methods generally integrate semantic analysis with grasp detection, resulting in low efficiency and poor generalization when dealing with new tasks and hardware. To address these problems, we propose SemSegGrasp, which reformulates TOG as a semantic segmentation problem based on point cloud and text matching. By decomposing TOG into semantic segmentation and grasp detection, SemSegGrasp can significantly enhance both the efficiency and generalization performance of TOG. Moreover, it can be combined with any off-the-shelf grasp detection algorithms in a plug-and-play manner. For semantic segmentation, SemSegGrasp first utilizes a Vision-Language Model (VLM) to generate local geometric descriptions of the target object. These descriptions are then fed into a Large Language Model (LLM) along with user instructions to obtain operational guidance. Subsequently, we separately encode the input point cloud and the operational guidance and obtain their features. Leveraging multi-head cross-attention, we conduct a matching process between these two types of features to predict the probability of each point serving as a TOG grasp point, i.e. semantic segmentation. Finally, the grasp pose is determined by fusing the segmentation results with the candidate poses generated by an existing grasp detection algorithm. Experimental results on the publicly available TaskGrasp dataset and a real-world setting show that our SemSegGrasp method achieves state-of-the-art performance, outperforming existing methods by at least 5% and 10% on new tasks, respectively.

Generating educational questions of fairytales or storybooks is vital for improving children's literacy ability.However, it is challenging to generate questions that capture the interesting aspects of a fairytale story with educational meaningfulness.In this paper, we propose a novel question generation method that first learns the question type distribution of an input story paragraph, and then summarizes salient events which can be used to generate high-cognitive-demand questions.To train the event-centric summarizer, we finetune a pre-trained transformer-based sequenceto-sequence model using silver samples composed by educational question-answer pairs.On a newly proposed educational questionanswering dataset FairytaleQA, we show good performance of our method on both automatic and human evaluation metrics.Our work indicates the necessity of decomposing question type distribution learning and event-centric summary generation for educational question generation.

Physical common sense plays an essential role in the cognition abilities of robots for humanrobot interaction.Machine learning methods have shown promising results on physical commonsense learning in natural language processing but still suffer from model generalization.In this paper, we formulate physical commonsense learning as a knowledge graph completion problem to better use the latent relationships among training samples.Compared with completing general knowledge graphs, completing a physical commonsense knowledge graph has three unique characteristics: training data are scarce, not all facts can be mined from existing texts, and the number of relationships is small.To deal with these problems, we first use a pre-training language model BERT to augment training data, and then employ constrained tucker factorization to model complex relationships by constraining types and adding negative relationships.We compare our method with existing state-ofthe-art knowledge graph embedding methods and show its superior performance.

Zhenjie Zhao, Andrew Cattle, Evangelos Papalexakis, Xiaojuan Ma. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Zhenjie Zhao, Xiaojuan Ma. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

When human partners attend to peripheral computing devices while interacting with conversational robots, the inability of the robots to determine the actual engagement level of the human partners after gaze shift may cause communication breakdown. In this paper, we propose a real-time perception model for robots to estimate human partners' engagement dynamics, and investigate different robot behavior strategies to handle ambiguities in humans' status and ensure the flow of the conversation. In particular, we define four novel types of engagement status and propose a real-time engagement inference model that weighs humans' social signals dynamically according to the involvement of the computing devices. We further design two robot behavior strategies (explicit and implicit) to help resolve uncertainties in engagement inference and mitigate the impact of uncoupling, based on an annotated human-human interaction video corpus. We conducted a within-subject experiment to assess the efficacy and usefulness of the proposed engagement inference model and behavior strategies. Results show that robots with our engagement model can deliver better service and smoother conversations as an assistant, and people find the implicit strategy more polite and appropriate.

In this paper, a new background modeling method called stacked multilayer self-organizing map background model (SMSOM-BM) is proposed, which presents several merits such as strong representative ability for complex scenarios, easy to use, and so on. In order to enhance the representative ability of the background model and make the parameters learned automatically, the recently developed idea of representative learning (or deep learning) is elegantly employed to extend the existing single-layer self-organizing map background model to a multilayer one (namely, the proposed SMSOM-BM). As a consequence, the SMSOM-BM gains several merits including strong representative ability to learn background model of challenging scenarios, and automatic determination for most network parameters. More specifically, every pixel is modeled by a SMSOM, and spatial consistency is considered at each layer. By introducing a novel over-layer filtering process, we can train the background model layer by layer in an efficient manner. Furthermore, for real-time performance consideration, we have implemented the proposed method using NVIDIA CUDA platform. Comparative experimental results show superior performance of the proposed approach.