Most recent Acoustic Word Embedding (AWE) systems utilize an autoencoder-like approach to compress speech features of arbitrary shapes into fixed-size numerical vectors and then reconstructing it, thereby capturing essential patterns in the data. Unfortunately, AWE models have commonly relied on supervised learning, necessitating extensive textual data, or have employed unsupervised dynamic-based methods that are computationally demanding. This paper introduces an unsupervised approach to AWE, leveraging a self-supervised Visual-Grounded Speech (VGS) model, eliminating the need for dynamic algorithms or textual data. Additionally, we propose a fine-grained ABX evaluation protocol that meticulously assesses the acoustic similarity between spoken segments, providing a more comprehensive and fair evaluation of model performance. Our findings indicate that proposed visual-grounded approach allows the AWE model to function in a truly unsupervised manner without relying on text data and computationally intensive dynamic-based algorithms, while also achieving performance comparable to other approaches.

Most computer-assisted pronunciation training (CAPT) systems for second language (L2) learners focus on detecting mispronunciation based on predefined phonemes and assigning pronunciation scores. However, these systems often lack visual feedback or detailed corrective guidance, limiting learners’ opportunities for significant improvement. This paper presents a key advance toward developing a CAPT system that offers detailed visual feedback on articulatory movements using real-time magnetic resonance imaging (rtMRI) articulatory landmarks. The limited availability of paired speech and articulatory landmark data, typically involving only a few speakers, poses a challenge for generalizing across diverse speech patterns. To address this, we propose leveraging pretrained wav2vec 2.0 embeddings, fine-tuned to generate articulatory contours mapped to xy coordinates based on rtMRI landmark data. As evaluated with the rtMRI USC-TIMIT dataset, our system effectively reconstructs visual articulatory movements from speech, marking a significant step toward enhanced visual pronunciation learning.

Vocabulary learning that incorporates visual information has become widely recognized as an alternative to context-based methods. However, few studies focus on learners' incorrect answers. On the Other hand, fossilization caused by repeated errors has been a concern. Our proposed system, L-VEIGe, effectively prevents repeated errors by visualizing learners' incorrect answers through image generation, which encourages introspection. However, there exists a 'Feature Disappearance' problem, where the generated images for incorrect answers lack sufficient information for comprehension. This study proposes a method for quantitatively evaluating these error images from a cognitive perspective.

Research surveys have always posed a challenge for novice researchers who lack research training. These researchers struggle to understand the directions within their research topic and the discovery of new research findings within a short time. One way to provide intuitive assistance to novice researchers is by offering relevant knowledge graphs $(KG)$ and recommending related academic papers. However, existing navigation knowledge graphs mainly rely on keywords or meta information in the research field to guide researchers, which makes it difficult to clearly present the hierarchical relationships, such as inheritance and relevance between multiple related papers. Moreover, most recommendation systems for academic papers simply rely on high text similarity, confusing researchers as to why a particular article is recommended. They may lack the grasp of important information about the insight connection between ‘Issue resolved’ and ‘Issue finding’ that they hope to obtain. This study aims to support research insight surveys for novice researchers by establishing a hierarchical tree-structured knowledge graph that reflects the inheritance insight and the relevance insight among multiple academic papers on specific research topics to address these issues.

With the exponential growth in the number of papers and the trend of AI research, the use of Generative AI for information retrieval and question-answering has become popular for conducting research surveys. However, novice researchers unfamiliar with a particular field may not significantly improve their interaction efficiency with Generative AI because they have not developed divergent thinking in that field. This study aims to develop an in-depth Survey Forest Diagram that guides novice researchers in divergent thinking about the research topic by indicating the citation clues among multiple papers to help expand the survey perspective for novice researchers.

Novice researchers often face difficulties in understanding a multitude of academic papers and grasping the fundamentals of a new research field. To solve such problems, the knowledge graph supporting research survey is gradually being developed. Existing keyword-based knowledge graphs make it difficult for researchers to deeply understand abstract concepts. Meanwhile, novice researchers may find it difficult to use ChatGPT effectively for research surveys due to their limited understanding of the research field. Without the ability to ask proficient questions that align with key concepts, obtaining desired and accurate answers from this large language model (LLM) could be inefficient. This study aims to help novice researchers by providing a fish-bone diagram that includes causal relationships, offering an overview of the research topic. The diagram is constructed using the issue ontology from academic papers, and it offers a broad, highly generalized perspective of the research field, based on relevance and logical factors. Furthermore, we evaluate the strengths and improvable points of the fish-bone diagram derived from this study's development pattern, emphasizing its potential as a viable tool for supporting research survey.

Characteristics of Highly Sensitive Persons (HSPs), such as “depth of processing,” “overstimulation,” “emotional reactivity and empathy,” and “sensitivity to subtleties,” often present challenges due to their high Sensory Processing Sensitivity (SPS) to environmental stimuli. This study investigates the differences in SPS among learners and the impact of various avatars on video presentations, a medium that has seen increased use due to COVID-19. We surveyed 20 participants who engaged with SDG instructional videos featuring four different avatars. Using the HSPS-J19 self-assessment tool, analysis of their SPS responses revealed a normal distribution of SPS scores, indicating individual differences. Additionally, correlations were found between HSPS-J19 scores and participants' impressions and motivation regarding avatar presentations. Cluster analysis results suggested that the group with a higher tendency towards HSP traits benefited more from appropriate avatar use. Based on these findings, we designed an online lecture support environment that allows for the control of video stimuli. This research explores an underexamined area and aims to enhance online lectures for HSPs, who constitute approximately 15% to 20% of the population. Supporting high SPS learners is particularly significant in the post-COVID-19 era.

STGs, a longstanding video game subgenre, have grown more intricate over time, deterring new players. To address this, a training system is required to improve character control skills in STG games. Bayesian Knowledge Tracing (BKT) is a common approach researchers use to monitor and assess students' progress. While BKT is effective in evaluating intellectual knowledge, it falls short in assessing character control skills in STG, a form of motion knowledge. This study proposes a Skill Tracing (ST) model that combines BKT approaches to monitor both cognitive knowledge and character control abilities. Results indicate its superiority in skill-tracking tasks over traditional BKT, offering a more accurate prediction of players' skill levels.

With the widespread use of the internet, it has become increasingly crucial to extract specific information from vast amounts of academic articles efficiently. Data mining techniques are generally employed to solve this issue. However, data mining for academic articles is challenging since it requires automatically extracting specific patterns in complex and unstructured layout documents. Current data mining methods for academic articles employ rule-based (RB) or machine learning (ML) approaches. However, using rule-based methods incurs a high coding cost for complex typesetting articles. On the other hand, simply using machine learning methods requires annotation work for complex content types within the paper, which can be costly. Furthermore, only using machine learning can lead to cases where patterns easily recognized by rule-based methods are mistakenly extracted. To overcome these issues, from the perspective of analyzing the standard layout and typesetting used in the specified publication, we emphasize implementing specific methods for specific characteristics in academic articles. We have developed a novel Text Block Refinement Framework (TBRF), a machine learning and rule-based scheme hybrid. We used the well-known ACL proceeding articles as experimental data for the validation experiment. The experiment shows that our approach achieved over 95% classification accuracy and 90% detection accuracy for tables and figures.

The final objective of this research project is to develop a lecture archive summarization platform that can extend learners' experience by automatically estimating and providing temporal and spatial ROI (Regions of Interest) according to multimodal features and learners' learning logs in lecture archives that record face-to-face lectures. To develop this platform, we (a) establish a method for extracting spatiotemporal multimodal features of lecture archives and (b) construct a method for estimating a learning style model based on the learning logs when watching the archives with the extracted features. Furthermore, to maximize the learning effect, we will (c) develop a prototype system to adaptively control the spatiotemporal ROI at the terminal side according to the learning style model as an adaptive summarization. This article describes the concept of the proposed platform and the initial analysis of learners' learning logs.

Due to the COVID-19 pandemic, most universities endeavored to adopt online education as an alternative to conventional face-to-face classroom instruction. However, capturing students' Temporal Region of Interest (T-ROI) in long-duration video lectures poses a significant challenge. Therefore, lecture archive summarization becomes essential from an online perspective. The results of lecture archive summarization still require further improvement. This research aims to distinguish T-ROI using a speech processing approach h. Our plan is divided into collecting instructors'/presenters' voice datasets, clarifying the T-ROIs through sound processing technology, and building a suitable deep neural network architecture to detect the T-ROIs in the actual lecture archives automatically. We will inevitably encounter various challenges to achieve the objective, such as individual differences. This article describes the experimental dataset collection design considering individual differences and lecture room environments. It summarizes how such efforts will be effective in realizing personalized voice style detection and improving the accuracy of speech processing in real environments.

Virtual reality technology has the potential to revolutionize immersive experiences in various applications, including office settings. However, efficient text entry in VR remains a significant challenge. This study addresses this challenge by proposing a machine learning-based solution, the 2S-LSTM typing method, to enhance text entry performance in VR. The 2S-LSTM leverages the back of the hand image. It employs a two-stream Long Short-Term Memory (LSTM) network, combined with a Kalman Filter (KF), to improve hand position tracking accuracy and reduce jitter. The results from questionnaire-based evaluations and typing data analysis demonstrate the superiority of the 2S-LSTM solution over existing solutions like Oculus Quest 2 and Leap Motion in terms of typing efficiency, fatigue reduction, accurate hand position replication, and positive user experience. These findings contribute to the advancement of text entry in VR environments and pave the way for immersive work experiences in the office and beyond.

In Web-based investigative learning, learners are expected to construct wider and deeper knowledge by navigating a great number of Web resources/pages. In elaborately investigating an initial question, learners are expected to decompose an initial question into related question to be further investigated. However, it is difficult for learners to conduct question decomposition in concurrence with their knowledge construction. In our previous study, we have proposed a model of Web-based investigative learning, and developed the system named interactive Learning Scenario Builder (iLSB for short). Although iLSB could promote self-directed investigative learning, learners often decompose a question into unrelated sub-questions. This suggests the necessity of promoting reflection on question decomposition by diagnosing the appropriateness of question decomposition. Toward this issue, we have proposed a method for diagnosing the appropriateness of question decomposition with Linked Open Data (LOD). In this paper, we describe an adaptive prompting with diagnosed results for reflection on question decomposition. This paper also reports a case study whose results suggest the potential for promoting reflection on improper question decomposition.

Most foreign students studying abroad lack of effective academic listening ability which is considered to be essential for them to achieve their academic successes. Moreover, as listening comprehension ability is also considered to be the most difficult to improve in contrast with the other three (Reading, Speaking and Writing), the purpose of this research is to support the training of academic listening skills for students pursuing academic success in a foreign educational institute. We have identified several learning strategies proved to be effective for cultivating academic listening skills from related work and built up the respective strategy models. Based on the established strategy models, we are now in the process of designing and developing various strategy objects (function units), so as the mashups environment where these objects can be assembled and operated. Unlike previous learning systems that provided identical functions to the learners, this research is expected to provide the learners with self-adjustable learning environment by putting together various strategy objects provided. We also attach semantic meanings (listening strategies and tactics) to each object to improve the metacognitive awareness of strategy application of the learners, for the effectiveness in listening practice as proved in past studies. Furthermore, a feedback agent is to be implemented to recommend proper strategy objects to the learners based on their learning situations. As a result, the learners are expected to be able to: practice their listening under an adaptive learning environment, strengthen their metacognitive awareness of the strategy application, and adjust their learning environment constantly with the support of the feedback agent or through peer reviews.

Disaster response works usually contain many problems, which need to be solved immediately. Most of such problems are consisted ethical matters. Therefore, it is essential for disaster responders having an awareness of ethical consideration to make decision accurately. To address such requirement, we designed a training game environment named MAGNITUDE. The game was proposed to improve non-technical skills, i.e. ethical decision-making. MAGNITUDE combined two types of game genre, simulation and role-playing game (RPG). By implementing simulation genre, it is expected to provide realistic-situation like disaster response. Whereas, by adopting RPG genre, we presume that MAGNITUDE encouraged a player to increase the level of his/her non-technical skill from a novice to an expert. In this paper, we explain the RPG pattern, which yields the ethical gameplay implemented in MAGNITUDE game.

The main issue addressed in this paper is how to improve the learning situation of self-directed learning on resource finding and organization from the Word Wide Web. In this paper, we have firstly proposed a multi-layer map model that visualizes basic learning behaviors when using the internet for locating and organizing learning resources. It provides learners with the structures of the found resources, the tools for their semantic management, and also an easy way to share the resources via the map representation. A system based on the proposed model has also been developed, that enables individual learners to easily locate suitable learning resources from the Web by referring resource maps and also to organize them as personal topic maps. By referring to a community topic map which merges all the personal topic maps created by individual self-directed learners, the learners can share their own resources and collect those of other learners into their learning topics. As a result, the learners re-organize their personal topic maps by taking the resource from the community topic map, and at the same time contribute to the community topic map through their personal topic maps. A case study conducted to evaluate the effectiveness of the system produced several positive results which validated our hypothesis.

In order to assist EFL students with low English proficiency in learning writing skill, we propose a framework of a virtual environment to evaluate common errors that the students often conduct in writing essay. The system is a service applied in platform linked with other NLP services to help with language analysis. The system mainly focuses on finding the writing errors related to semantic meaning selection, incorrect structure to indicate the intended meaning, non-smoothing sentence in topic, and etc. The system improves the students' writing skill by providing questions relating to the matters they are writing.

Many responses of catastrophic natural disaster did not perform properly to an appropriate standard. This often occurred when first responders were involved, especially novice volunteer who did not have the accurate decisionmaking skill. One of the main issues is the lack of regular training to develop such skills. It has been pointed out that exercise of the non-technical abilities, such as decision-making has an enormous impact on effective disaster response. However, some researches show that there are difficulties to conduct live practice for the disaster situation similarly. In addition, the novice volunteer cannot receive maximum advantages from live training due to feedback limitation where reflection from actual circumstances is required to improve those skills.The purpose of this research is to design a mobile game based learning (mobile GBL) for developing such skills. First of all, we conducted a preliminary survey to assess the awareness of the ethical decision-making skill of the novice volunteer from high school and university organizations in Indonesia. We asked these respondents to answer three categories of questions encompassed six components of moral intensity. We also interviewed some experts from the official search and rescue (SAR) organization in Indonesia to confirm first responder requirements. Based on these preliminary surveys and interviews, we have designed a training system called Magni tude which enables the novice volunteer to develop their ethical decision making skill at all times during official disaster management training inside and outside of class, and expect them to improve their performance in disaster response activities.

The main topic addressed in this paper is to help a novice graduate/undergraduate student compose his/her presentation document by means of presentation schema that represents heuristics for presenting research contents to be shared by laboratory members. The key idea is to propose a model of presentation structure, which represents roles of and sequences among presentation slides included in the documents with metadata. Following this model, the presentation schema is defined as a typical presentation structure for the laboratory members. This paper accordingly introduces a technique based on association rule mining for automatically extracting the presentation schema from the repository of the documents accumulated in the laboratory. In addition, we report case studies for investigating how to configure the thresholds of the mining and how the schema extracted is valid in comparing the ones between different laboratories.

In order to properly create presentation documents as research activity, it is necessary to get and accumulate experiences in composing the semantic structure that represents what to present and how to sequence the contents presented. However, it is not easy for novice researchers because they have fewer experiences in creating presentation documents. This paper proposes scaffolding for composing the semantic structure of presentation documents with presentation schema that is typical semantic structure embedded in the presentation documents accumulated in a research group. The results of a case study suggest that the schema-based scaffolding contributes to creating presentation documents.

Graduate and undergraduate students in the laboratory usually deal with not only formal information such as research articles and presentation documents but also informal information which represents a process of research activities. However, it is difficult for new students belonging to the laboratory to acquire such informal information from researchers and other students through the laboratory life. In order to resolve this issue, we have developed an article revising support system called CommentManager that facilitates the process of article revising and extracts the knowledge for revising from the informal information accumulated from the article revising processes of the laboratory members.

The main topic in this paper is how to effectively help research group members share and reuse presentation documents. The key idea is to propose a presentation semantics framework, which represents semantic roles of and relations among presentation slides with metadata. We then discuss a machine learning technique for analyzing the semantics roles and relationships automatically from the repository of the documents accumulated in the research group. This paper also demonstrates interactive Web services that recommend the metadata to be attached to the documents newly made, and that diagnose the presentation semantics of the documents.

The main topic addressed in this paper is to support researchers and students to make, share, and reuse their presentation documents. The presentation documents are well-organized information for their research. However, it is not so easy to share and reuse such documents since the semantic structure of the documents are not explicitly represented. In order to resolve this issue, we propose a presentation semantics model which represents roles and relations among the presentation objects in the presentation document. We also introduce a presentation support service for editing the presentation semantics and for designing a new presentation document as a Web 2.0 service.

The main topic addressed in this paper is how to help learners select some instructive hypermedia-based learning resources according to their learning contexts from the Web. Our approach is to provide a digital library for web-based learning called e-Learning Library, which includes learning resource repository, local indexing, and adaptive navigation support. This aims to promote their learning with diverse learning resources involving a certain topic.