This study examines the asynchrony detection ability between auditory and vibrotactile stimuli to improve the "VIBES" singing timing support system for Deaf and Hard-of-Hearing (DHH) individuals. We conducted synchrony judgment tasks with 20 DHH participants using a single device to present auditory and vibrotactile stimuli. We tested two conditions: a regular rhythm (RR) condition with contextual beats (120 beats per minute) surrounding the target stimulus and a no-rhythm (NR) condition where we presented only the target stimulus. Our results showed a significant negative shift in the point of subjective simultaneity (PSS: the timing at which two different sensory stimuli are perceived as co-occurring) under the NR condition (-15 [ms]), indicating that presenting vibrotactile stimuli before auditory stimuli facilitates simultaneity perception. We observed more expansive temporal binding windows (TBW: the period during which stimuli from different senses are integrated and perceived as a single event) when auditory stimuli preceded vibrotactile stimuli. This suggests that unintended vibrotactile interference from auditory stimuli occurs when using a single device. We also noted individual differences in asymmetry between the TBW of RR and the NR conditions. These findings suggest DHH participants may detect asynchrony based on comparison with surrounding stimuli rather than direct comparison of stimulus pairs, highlighting the importance of accurate timing estimation for adequate singing support.

With the advancement of hearing aids and cochlear implants, auditory perception among the Deaf and Hard-of-Hearing (D/HoH) people has improved, including the recognition of environmental and instrumental sounds. However, even with these improvements, it remains difficult for many deaf individuals to perceive emergency vehicle sirens. This study proposes more perceivable siren patterns by modifying pitch and tempo. Through awareness surveys and listening experiments with D/HoH individuals, we examined the perceived urgency and audibility of modified sirens. The results suggest that increasing the tempo improves both recognizability and perceived urgency, while higher octave shifts tend to be rated as noisy or annoying.

Abstract This paper addresses the issue in live events, especially in sports viewing, where individuals who are deaf, hard of hearing (DHH), blind, or have low vision (BLV) struggle to access sufficient information. Traditionally, information accessibility has relied on specific professionals or volunteers, which is often inadequate. To tackle this challenge, we propose a mechanism that facilitates information sharing for all individuals, regardless of their abilities. We have developed an inclusive web application tailored to address these needs. This application is beneficial not only for DHH and BLV individuals but also for all audiences. Additionally, the system’s applicability extends to other domains, such as museum visits. This paper details the newly developed web application and presents the outcomes of pilot studies conducted in sports viewing and museum settings with DHH and BLV participants. The results of these experiments are analyzed to assess the system’s effectiveness and to identify future improvement areas.

Abstract Access to music for people who are Deaf/Hard of Hearing (D/HoH) includes the instrumental portion and the lyrics. While closed captioning can provide the lyrics in text format, it does not necessarily provide accurate timing of the lyrics with the instrumental portion. This study aims to clarify how vibrotactile stimuli affect understanding the onset timing of song lyrics for D/HoH people. To achieve this goal, we developed a system called VIBES: VIBrotactile Engagement for Songs that simultaneously provides music and vibration playback as an iPhone app. Unlike other vibrotactile systems for music, which focus primarily on the percussion/beat or frequencies of the instrumental portions, VIBES presents vibrations for the timing of vocal utterances, syllable by syllable. We conducted a study with 10 participants to determine the system’s effectiveness by comparing the understanding of lyric timing with vibration and sound only. Statistically, one of the four songs in the experiment showed significant differences, where understanding lyric timing with vibration is better than in sound-only conditions. Two songs revealed the opposite results, and the other showed no differences between the two conditions. Although these results were unexpected, we obtained several findings to make the next version of VIBES, such as the frequency of vibrations.

Some deaf and hard-of-hearing (DHH) individuals like to “listen to” music. For them to access a wider range of music, both the preference for and the audibility of music are important. As a starting point for understanding music accessibility, we investigated the audibility and preference for 19 musical instruments using five pitches totaling 95 timbres. To examine how people listen to music daily, owing to the difficulty of having participants come to the laboratory due to the spread of COVID-19, we conducted the experiment online. Participants listened to short clips of 95 sounds and subjectively answered questions on their audibility and preferences. An analysis of their subjective answers revealed the audibility and preference differences among instruments, specifically that the recorder lacks the same audibility and preference as some of the other instruments at certain pitches. However, there was little correlation between audibility and timbre preferences. The study findings will be used in music classes in which DHH students participate and indicate that timbre audibility is necessary for building music recommendation systems for DHH individuals. We also discuss the limitations and possibilities of online experiments using sound. Music, timbre, deaf and hard of hearing, audibility, preference, accessibility

We have promoted a project called "Information Support of Everyone, by Everyone, for Everyone (ISeee). "We have been developing a live timeline posting/reading system to use at sports games for audiences to share data consisting of photos and texts. We describe two experiments using the system for the purpose of information support for deaf and hard-of-hearing people (DHH). The results suggest that the data generated with the live timeline system is helpful as information support for DHH.

We previously investigated how hearing-impaired people perceive music in several types of musical experiments. By following-up on the results of some of these experiments with a single test subject Sd, we found that the ability of the subject to perceive music was high and that she appreciated music in a way that was different from that of hearing people. In this paper, we describe three musical experiments with hearing-impaired subjects, their results, and Sd's music perception through the experiments. The three experiments involved the Music Puzzle game, the appreciation of harmony, and tempo perception. Music Puzzle is a music game we made that is played on a tablet and is intended to be used by hearing-impaired persons as a serious game with which they can improve their hearing ability by continuously playing it. The experiment on appreciation of harmony was conducted with three subject groups, and the result showed that experience with music affected the appreciation of music accompanied with the tonal code. Tempo perception was investigated with a simple game in which the subjects tapped along with the tempo of the music. By examining the subjects' hearing acuity in standard medical hearing tests and crossing over the results of these musical experiments, we observed that hearing acuity is not necessarily related to the perception and understanding of music.

In this paper, we describe an experiment on the cognition of harmony done by three groups: people who have hearing-impairments, people who have little experience of playing music, and people who have a lot experience of playing music. By modifying the accompaniment key, we prepared five types of harmony for a single melody, provided them to the subjects, and asked for their preferred harmony. Their preferences for the accompaniments differed. People with music experience strongly preferred the original accompaniment, while hearing-impaired people cared little about whatever the harmony was. Two hearing groups' preferences did not differ much for the original and other accompaniments, but people with little musical experience had the preferences they lay between those of the other two subject groups. Though many hearing-impaired people are fond of music, this experiment on the cognition of harmony did not find that they appreciate musical harmony. The results of the different preferences of people with a little and a lot of musical experience suggest their preferences may differ between in other aspects of music.

This paper describes an experiment on the rhythm perception by hearing-impaired people using three musical contents: vocal-based songs, vocal-only songs, and instrumental music. The real world consists of many kinds of sounds, not only human speech but also music, songs, and environmental noises. Although typical listening ability exercises these days are primarily optimized for perception of speech rather than other sounds, we assume that musical features can help hearing-impaired people to segregate information sounds and the other sounds. Our project is intended to create a musical training that enhances hearing-impaired people's listening ability. In this study, to begin the project, we investigated how the hearing-impaired people recognize the rhythmical features in listening to the songs. Nine hearing-impaired undergraduates participated in the tapping experiment. Experimental results showed that musical experience significantly improved their rhythm perception.

There are many hearing-impaired college students who have interest in music and some of them listen to music every day. What do hearing-impaired people enjoy in music? Among the several elements in music, focus was placed on timbre in an experiment. Timbre differs if a player plays a musical score with different musical instruments. Several sets of music data were prepared from commercial CDs in which the same phrases were played with different musical instruments and subjects were asked to listen to and compare the timbres. Subject groups consisted of hearing-impaired people, hearing people with little or no experience in playing music, and hearing people with active experience in playing music. Significant differences were found in the ability to differentiate timbre among the three groups. Hearing-impaired people were found to be good at noticing sets in which all timbres were similar by comparison but they encountered difficulties in differentiating timbres in other cases. The brightness curve was found to be a candidate as an attribute to explain these difficulties.

Emotion recognition in polygons and curved shapes is described. Attneave-style polygons that have no specific meaning by themselves are generated following an algorithm. Attneave-style closed polygons and curved shapes that are transformed with the spline function from the polygons are used to see whether the two types of shapes have differences in emotion recognition. Subjects were found to recognize emotions in curved shapes more than they did in polygons. The kinds of emotions recognized in each type of drawing were different-subjects recognized mostly anger in polygons but joy in curved shapes. They least recognized anger in curved shapes. There were no specific emotion recognition relationships between the pairs of polygons and their transformed curved shapes.

We are building a musical performance assistance system that uses visual information to supplement the musical communication among deaf and hard-of-hearing people playing music together, enabling them to enjoy music communication playing music representing a certain emotion. We previously collected drawings for which the drawer had an emotion in mind in order to improve the communication in our system, which presents them to the players to enable them to focus on expressing a specific emotion. We thus need to better understand the properties of images that relate to an emotion. After computing the principal components of the image properties, we clustered the drawings into groups representing one of four emotions. We also asked viewers to categorize them on the basis of the emotion they elicited in the viewer. The results showed that the viewers based their judgment on the specific meaning of the drawing rather than the shape, if the shape of drawings was not abstract. The drawings intended to represent fear were well grouped by clustering, while the viewers categorized them as either fear or sadness. The results imply that we can add new drawings with intended emotions to our system as long as their image properties meet our analysis.

We have conducted a series of experiment to investigate the potential of music communication based on emotion for hearing-impaired people and people with hearing abilities. The purpose of the experiments is to design a performance assistance system to enable hearing-impaired people to enjoy taking part in a music ensemble. In this paper, we describe an experiment on recognizing emotion in two media: moving images of a professional percussionist giving an improvised performance and the sounds extracted from the performance. The experiment was designed to investigate whether the image of a performance is important in recognizing emotion in the performance. The result showed that there was no significant difference in recognition of emotion between the two media. Our results were consistent with those of previous experiments: (1) there was no significant difference in recognizing emotion between hearing-impaired people and people with hearing abilities, and (2) ldquofearrdquo was the least recognized of the four emotions.

We conducted an experiment to determine the abilities of hearing-impaired and normal-hearing people to recognize intended emotions conveyed in four types of stimuli: a drum performance, a drum performance accompanied by a drawing expressing the same intended emotion, and a drum performance accompanied by one of two types of motion pictures. The recognition rate was the highest for a drum performance accompanied by a drawing even though participants in both groups found it difficult to identify the intended emotion because they felt the two stimuli sometimes conveyed different emotions. Visual stimuli were especially effective for performances whose intended emotions were not clear by themselves. The difference in ability to recognize intended emotions between the hearing-impaired and normal-hearing participants was insignificant. The results of this and a series of experiments will enable us to better understand the similarities and differences between how people with different hearing abilities encode and decode emotions in and from sound and visual media. We should then be able to develop a system that will enable hearing-impaired and normal-hearing people to play music together.

With the purpose of building a performance assistant system with visual cues for hearing-impaired people and people with normal hearing abilities to play music together, we need to better understand how hearing-impaired people express and understand music with visual assistance. We have conducted a series of experiments to determine how hearing-impaired people understand an emotion in a drum performance from its intended emotion. These experiments showed that communication based on emotion through musical performance is possible. Therefore we need to know the practical applications for a visual interface. In this paper, we describe an experiment that determines how hearing-impaired people understand an emotion in small simple drawings from its intended emotion. The results suggest that there are similarities in recognizing emotions between hearing-impaired people and people with normal hearing abilities.

We present a new method for visualizing musical expressions with a special focus on the three major elements of tempo change, dynamics change, and articulation. We have represented tempo change as a horizontal interval delimited by vertical lines, while dynamics change and articulation within the interval are represented by the height and width of a bar, respectively. Then we grouped local expression into several groups by k-means clustering based on the values of the elements. The resulting groups represented the emotional expression in a performance that is controlled by the rhythmic and melodic structure, which controls the gray scale of the graphical components. We ran a pilot experiment to test the effectiveness of our method using two matching tasks and a questionnaire. In the first task, we used the same section of music, played by two different interpretations, while in the second task, two different sections of a performance were used. The results of the test seem to support the present approach, although there is still room for further improvement that will reflect the subtleties in performance.

We propose a graphical method to express what a music performance is intended to create in the minds of the audience. Our graphical approach overcomes the problems associated with verbal labeling. Besides playing a melody and harmony, a music player tries to produce certain feelings in the audience by manipulating tempo, rhythm, articulation and dynamic changes. Despite the linear nature of music, the produced mood does not necessarily preserve the temporal sequence, and is mentally representable in different forms. As a first approximation to such a representation, we have developed a method involving a plane on which the musical expression elements are projected. First, expression elements for all notes in a musical section are derived. They are then arranged according to the importance of notes in consideration of the musical structure.

No abstract available.

This paper describes performance visualization with its purposes, significance, and our prototype system. Performance visualization provides us with a new user interface for music systems, a non-subjective explanation and judgment for musical expression, and a visual aid to understand, analyze, and compare performances and their musical structures. It will also introduce a more challenging research on designing a musical data mining interface not by tags nor by contents but by the mood. Other challenges include finding qualitative characteristics of music with analysis methods for information visualization and synesthesia between visualized and sonified information. We will describe which information should be shown on the visualized figures for expressive performance. Then we will show our prototype system to visualize expressive performance consisting of several instruments.

For most of the time, we enjoy and appreciate music performances as they are. Once we try to understand the performance not in subjective terms but in an objective way and share it with other people, visualizing the performance parameters is indispensable. In this paper, a figure for visualizing performance expressions is described. This figure helps people understand the cause and position of the performance expression as it has expressive cues, which coincide with the cognitive meaning of musical performance, and not by using only MIDI parameter values. The differences we hear between performances are clarified by visualized figures.