Demo Data from VLDB 2026 / reviewers in the wild / expert
Jaakko Hakulinen
dblp:25/4928 · also Jaakko Samuli Hakulinen
· DBLP profile ↗
57ranked-venue papers
6as first author
0since 2021 · last 2020
0000-0001-8054-7265ORCID · verified
Domains — the database's venue-derived domains; a paper can count in several
Human-computer interaction and ubiquitous computing · 32 · 3 first-authorGraphics, computer vision, multimedia, augmented reality and games · 26 · 5 first-authorArtificial intelligence and machine learning · 25 · 3 first-author
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Human-computer interaction and pervasive computing
5 papers |
Immersive interaction · 35% Wearable and physiological sensing · 12% Usability and user experience research · 12% | |
| Computer graphics and multimedia
2 papers |
Virtual and augmented reality · 60% Visualization and visual analytics · 40% |
Topics — the 11 heaviest of 15, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Usability and user experience research › evaluation methodology
AR evaluation |
0.4 | 1 | 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance · CHI 2020 |
Immersive interaction
augmented reality |
0.4 | 1 | 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance · CHI 2020 |
Immersive interaction › augmented reality
augmented reality simulation |
0.4 | 1 | 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance · CHI 2020 |
Wearable and physiological sensing
eye tracking |
0.4 | 1 | 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance · CHI 2020 |
Immersive interaction
virtual reality |
0.4 | 1 | 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance · CHI 2020 |
Virtual and augmented reality
immersive interaction |
0.4 | 1 | 2019 | The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos · VR 2019 |
Visualization and visual analytics
usability and user experience research |
0.4 | 1 | 2019 | The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos · VR 2019 |
Ubiquitous computing and smart environments › public displays
public display deployment |
0.3 | 1 | 2017 | Challenges in Public Display Deployments: A Taxonomy of External Factors · CHI 2017 |
Health and well-being technologies › autism therapy
autism intervention technology |
0.2 | 1 | 2016 | Promoting Joint Attention with Computer Supported Collaboration in Children with Autism · CSCW 2016 |
Collaborative and social computing
computer-supported cooperative work |
0.2 | 1 | 2016 | Promoting Joint Attention with Computer Supported Collaboration in Children with Autism · CSCW 2016 |
Virtual and augmented reality › immersive display
head-mounted display |
0.1 | 1 | 2019 | The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos · VR 2019 |
Methods — techniques the papers use, named apart from their topics
comparative user study · 0.6WebVR · 0.6survey · 0.4iterative design · 0.4user study · 0.4exploratory research · 0.3case study · 0.3mixed-methods evaluation · 0.2gesture-based interaction · 0.2
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2020 | Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial MaintenanceabstractAugmented reality (AR) presents a variety of possibilities for industrial maintenance. However, the development of real-world AR solutions has been limited due to the technological capabilities and uncertainty with respect to safety at deployment. We introduce the approach of using AR simulation in virtual reality (VR) coupled with gaze tracking to enable resource-efficient AR development. We tested in-field AR guidance and safety awareness features in an iterative development-evaluation process with experts from the elevator maintenance industry. We further conducted a survey, utilizing actual gaze data from the evaluation to elicit comments from industry experts on the usefulness of AR simulation and gaze tracking. Our results show the potential of AR within VR approach combined with gaze tracking. With this framework, AR solutions can be iteratively and safely tested without actual implementation, while gaze data provide advanced objective means to evaluate the designed AR content, documentation usage, and safety awareness. Alisa Burova, John Mäkelä, Jaakko Hakulinen, Tuuli Keskinen, Hanna Heinonen, Sanni Siltanen, Markku Turunen |
CHI | 3 |
| 2019 | Exploring Globally Inclusive Online Collaboration for Indian and Finnish SchoolchildrenabstractWe present our exploratory work on globally inclusive online collaboration for schoolchildren in India and Finland, using CityCompass, an online virtual navigational application for conversational language learning. User studies with Indian children, who collaborated with a Finnish researcher, showed several barriers towards communication and collaboration, including issues from limited access to computers and gaming to socio-cultural effects of a large power distance and face-saving. By adding a dramatized scenario, the Bollywood Method, these barriers were reduced. Next, we replicated the study with Finnish children, who collaborated with an Indian researcher. Here, previous computer and gaming experience reduced the motivation towards communication. In this paper, we present studies conducted in Finland, compare it with the Indian ones, and discuss opportunities for inclusive collaboration between Indian and Finnish children. Overall, our findings indicate that online collaboration is affected by differences in computer skills, video gaming experience, and socio-cultural communication norms. Sumita Sharma, Pekka Kallioniemi, Jaakko Hakulinen, Tuuli Keskinen, Markku Turunen |
IDC | 3 |
| 2019 | CityCompass VR - A Collaborative Virtual Language Learning Environment
Pekka Kallioniemi, Kimmo Ronkainen, Jussi Karhu, Sumita Sharma, Jaakko Hakulinen, Markku Turunen |
INTERACT (4) | 5 |
| 2019 | Promoting local culture and enriching airport experiences through interactive storytellingabstractExperiences in airports may shape future travel plans and contribute to tourism destination development. However, a chaotic environment and time-consuming procedural routines in airports may result in negative associations towards the host country and its culture. Despite the existence of assistive airport applications, little attention is given to facilitating travelers' engagement with cultural exploration. Alisa Burova, Chelsea Kelling, Tuuli Keskinen, Jaakko Hakulinen, Pekka Kallioniemi, Heli Väätäjä, Markku Turunen |
MUM | 4 |
| 2019 | The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videosabstract360° videos can be viewed in an immersive manner with a head-mounted display (HMD). However, it is unclear how the viewing experience is affected by basic properties of 360° videos, such as how high they are recorded from, and whether there are people close to the camera. We conducted a 24-participant user study where we explored whether the viewing experience is affected by A) camera height, B) the proximity and actions of people appearing in the videos, and C) viewer position (standing/sitting). The results, surprisingly, suggest that the viewer's own height has little to no effect on the preferred camera height and the experience. The most optimal camera height situates at around 150 centimeters, which hits the comfortable height range for both sitting and standing viewers. Moreover, in some cases, people being close to the camera, or the camera being very low, has a negative effect on the experience. Our work contributes to understanding and designing immersive 360° experiences. Tuuli Keskinen, Ville Mäkelä, Pekka Kallionierni, Jaakko Hakulinen, Jussi Karhu, Kimmo Ronkainen, John Mäkelä, Markku Turunen |
VR | 4 |
| 2018 | Overcoming socio-technical challenges for cross-cultural collaborative applicationsabstractOur research focuses on facilitating access to cross-cultural collaborative applications for schoolchildren. In this paper, we present two user studies with students from an underprivileged region in Delhi. In the first study, we found that Indian students were hesitant when using the application as compared with Finnish students in a similar study, for example. We purport these challenges were social - related to face-saving and power-distance - and technical - related to lack of access to and experience with computers. In the second study, we presented a dramatized scenario before the task to address the socio-technical challenges. Our findings suggest that adding a dramatized scenario facilitated meaningful collaboration by (a) overcoming the lack of previous computer and gaming experience and (b) reducing social and cultural barriers towards communication. We believe these findings extend to cross-cultural collaboration between students with varying computer experience. Sumita Sharma, Pekka Kallioniemi, Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Tuuli Keskinen |
IDC | 4 |
| 2018 | 'Wow! You're Wearing a Fitbit, You're a Young Boy Now!": Socio-Technical Aspirations for Children with Autism in IndiaabstractIn this paper, we build a case for incorporating socio-technical aspirations of different stakeholders, e.g. parents, care-givers, and therapists, to motivate technology acceptance and adoption for children with autism. We base this on findings from two studies at a special school in New Delhi. First, with six children with autism, their parents and therapists we explored whether fitness bands motivate children with autism in India to increase their physical activity. Second, with five parents and specialists at the same school, we conducted interviews to understand their expectations from and current usage of technology. Previous work defines a culture-based framework for assistive technology design with three dimensions: lifestyle, socio-technical infrastructure, and monetary and informational resources. To this framework we propose a fourth dimension of socio-technical aspirations. We discuss the implications of the proposed fourth dimension to the existing framework. Sumita Sharma, Krishnaveni Achary, Harmeet Kaur, Juhani Linna, Markku Turunen, Blessin Varkey, Jaakko Hakulinen, Sanidhya Daeeyya |
ASSETS | 7 |
| 2017 | Omnidirectional Video in Museums - Authentic, Immersive and Entertaining
Jaakko Hakulinen, Tuuli Keskinen, Ville Mäkelä, Santeri Saarinen, Markku Turunen |
ACE | 1 |
| 2017 | Challenges in Public Display Deployments: A Taxonomy of External FactorsabstractPublic display deployments are often subjected to various surprising and unwanted effects. These effects are frequently due to external factors properties and phenomena that are unrelated to the deployment. Therefore, we conducted a literature review within the public display domain to investigate the causes behind the reported issues. This work presents a taxonomy of external factors affecting deployments, consisting of six categories: weather, events, surroundings, space, inhabitants, and vandalism. Apart from a few positive examples, we predominantly found negative effects arising from these factors. We then identified four ways of addressing the effects: ignoring, adapting, solving, and embracing. Of these, ignoring and adapting are substantially more frequent responses than solving and embracing emphasizing the need for researchers to adapt. We present real-world examples and insights on how researchers and practitioners can address the effects to better manage their deployments. Ville Mäkelä, Sumita Sharma, Jaakko Hakulinen, Tomi Heimonen, Markku Turunen |
CHI | 3 |
| 2017 | Utilizing Experience Goals in Design of Industrial SystemsabstractThe core idea of experience-driven design is to define the intended experience before functionality and technology. This is a radical idea for companies that have built their competences around specific technologies. Although many technology companies are willing to shift their focus towards experience-driven design, reports on real-life cases about the utilization of this design approach are rare. As part of an industry-led research program, we introduced experience-driven design to metal industry companies with experience goals as the key technique. Four design cases in three companies showed that the goals are useful in keeping the focus on user experience, but several challenges are still left for future research to tackle. This exploratory research lays ground for future research by providing initial criteria for assessing experience design tools. The results shed light on utilizing experience goals in industrial design projects and help practitioners in planning and managing the product design process with user experience in mind. Virpi Roto, Eija Kaasinen, Tomi Heimonen, Hannu Karvonen, Jussi P. P. Jokinen, Petri Mannonen, Hannu Nousu, Jaakko Hakulinen, Pertti Saariluoma, Tiina Kymäläinen, Tuuli Keskinen, Markku Turunen, Hanna Koskinen |
CHI | 8 |
| 2017 | Guidelines for Designing Interactive Omnidirectional Video Applications
Santeri Saarinen, Ville Mäkelä, Pekka Kallioniemi, Jaakko Hakulinen, Markku Turunen |
INTERACT (4) | 4 |
| 2017 | Usage Data Analytics for Human-Machine Interactions with Flexible Manufacturing Systems: Opportunities and ChallengesabstractAnalyzing data from complex production systems and processes can be used in improving existing products, processes, and services, and innovating novel offerings. We report the findings from a six-month case study with a company developing flexible manufacturing systems. During a collaborative development process of a data analytics and visualization tool, our goal was to identify potential metrics, business opportunities, and challenges when utilizing logged data of end-users' human-machine interactions in development activities. Our key contributions include a characterization of the potential usage data metrics to be logged and visualized, identification of opportunities this data entails for business, and discussion about the challenges related to usage data logging in the studied context. Finally, we propose topics that should be considered in the organization before investing in usage data logging in the context of flexible manufacturing systems. Jari Varsaluoma, Heli Väätäjä, Tomi Heimonen, Katariina Tiitinen, Jaakko Hakulinen, Markku Turunen, Harri Nieminen |
IV | 5 |
| 2017 | Effect of gender on immersion in collaborative iODV applicationsabstractInteractive omnidirectional video (iODV) is a media format that allows the user to explore and interact with a 360-degree view of the recorded scenery. Recently, novel collaborative applications for presenting iODV content have emerged. Often their goal is to offer as immersive as possible experience to the users. Previous studies suggest that gender affects the §of immersion in virtual environments and other media, but there has been only little research on immersion in iODVs, and nothing in the context of collaborative interaction. In this research, we studied gender effect with participants (N=30, 15 pairs) performing a collaborative wayfinding task. Subjective data gathered with a customized immersion questionnaire showed statistically significant differences between male and female participants in Spatial Immersion and Involvement subscales. There were no statistical differences in Interaction, Realness, Physical and Auditory subscales. Several possibly affecting factors were observed during the task completion. Our results also indicate that performing interactive, collaborative tasks in iODV applications helps building a shared understanding between the users. Pekka Kallioniemi, Tuuli Keskinen, Jaakko Hakulinen, Markku Turunen, Jussi Karhu, Kimmo Ronkainen |
MUM | 3 |
| 2017 | Interaction with WebVR 360° video player: Comparing three interaction paradigmsabstractImmersive 360° video needs new ways of interaction. We compared three different interaction methods to find out which one of them is the most applicable for controlling 360° video playback. The compared methods were: remote control, pointing with head orientation, and hand gestures. A WebVR-based 360° video player was built for the experiment. Toni Pakkanen, Jaakko Hakulinen, Tero Jokela, Ismo Rakkolainen, Jari Kangas 0001, Petri Piippo, Roope Raisamo, Marja Salmimaa |
VR | 2 |
| 2016 | Collaborative Conversational Language Learning with CityCompassabstractCityCompass promotes collaborative conversational language learning by means of way-findings tasks situated in panoramic views of a city. Two remotely located participants have to navigate these 360 city panoramas as a tourist or a guide, to arrive at a preassigned destination. The application supports three modes of interaction and immersion: traditional mouse and keyboard interaction, embodied interaction using the Microsoft Kinect sensor and a virtual reality supporting 360 degree video panoramas on a mobile phone placed inside virtual reality googles (Samsung Gear VR). Pekka Kallioniemi, Sumita Sharma, Jaakko Hakulinen, Markku Turunen |
IDC | 3 |
| 2016 | Gesture-based Interaction for Individuals with Developmental Disabilities in IndiaabstractGesture-based interaction provides a multitude of benefits to individuals with disabilities, for example, enhancing social, motor and cognitive skills. However, applications that encourage self-efficacy by promoting a life-skill through simulations of real world scenarios are largely missing. We explore the benefits of using a gesture-based application for individuals with developmental disabilities. The context is a special school in New Delhi, Nai Disha, where we designed and developed an application, Kirana, that integrates arithmetic and social interaction to teach purchasing of items from a local grocery store. In our study, 18 participants with developmental disabilities, previously unable to visit a grocery store, used Kirana for three weeks. Our results indicate that gesture-based applications can teach a life skill and enable self-efficacy for individuals with developmental disabilities by breaking down complex tasks that require social, mathematical and decision-making skills. Sumita Sharma, Krishnaveni Achary, Blessin Varkey, Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Nitendra Rajput |
ASSETS | 6 |
| 2016 | Kirana: A Gesture-based Market App for Life Skills Learning for Individuals with Developmental DisabilitiesabstractKirana is a gesture-based application that simulates purchasing experience at a local Indian grocery store. It provides individuals with developmental difficulties a safe and controlled environment to explore the processes of purchasing an item: namely, deciding items to buy (decision making), calculating costs and balance (mathematical skills), and interactions via pointing (social interaction). Previous research has established that gesture-based interaction has the potential to enhance social, motor and cognitive skills for individuals with developmental disabilities. Currently, translating learnings from gesture-based virtual applications to real world scenarios is underexplored. Kirana is an approach to simulating practical real interactions, by breaking down complex tasks that require social, mathematical and decision-making skills, and encouraging self-efficacy. Skills learnt by purchasing items with Kirana, are potentially transferable to a real world store. Sumita Sharma, Krishnaveni Achary, Blessin Varkey, Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Nitendra Rajput |
ASSETS | 6 |
| 2016 | Promoting Joint Attention with Computer Supported Collaboration in Children with AutismabstractThere exists mounting evidence in favor of computer supported autism interventions at the individual level. However, the potential benefits of using computer supported collaboration to encourage social interactions between individuals with autism and typically developed individuals are underexplored, particularly in developing regions. We present an exploratory study of a collaborative gesture-based application, Balloons. The application encourages joint attention, which is defined as the shared attention between two individuals towards the same object. Using mixed methods, we evaluated Balloons for three weeks in New Delhi with ten medium-low functioning autistic children. Our findings suggest that employing CSC interventions for children with autism in India provide (a) observable improvements in social interaction with typically developed peers, (b) the opportunity to customize and individualize intervention to cater to a large spectrum of children and (c) the potential opportunity of reducing fears of certain objects. Sumita Sharma, Krishnaveni Achary, Blessin Varkey, Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Nitendra Rajput |
CSCW | 6 |
| 2016 | Evaluating Future Automation Work in Process Plants with an Experience-Driven Science Fiction PrototypeabstractThis paper introduces a video-illustrated science fiction prototype "Remote operator's day in a future control center in 2025" aimed at discovering opportunities for new interaction methods and smart automation for the future factories of Industry 4.0. The theoretical objective was to carry out experience design research in an industrial work context based on explicit user experience (UX) goals. This paper firstly describes how the nominated UX goals were implemented in a video-illustrated science fiction prototype and, secondly, how the prototype was evaluated in two research setups: as video sequences embedded in a Web survey, and as interviews carried out with process control workers. The main contribution is to demonstrate how the science fiction prototyping method can employ video-illustration as a means for future-oriented UX research, and how complementary user-centered methods may be used for validating UX goals by means of a science fiction prototype. Tiina Kymäläinen, Eija Kaasinen, Maiju Aikala, Jaakko Hakulinen, Tomi Heimonen, Hannu Paunonen, Jouni Ruotsalainen, Lauri Lehtikunnas, Petri Mannonen |
Intelligent Environments | 4 |
| 2015 | Evaluating a Future Remote Control Environment with an Experience-Driven Science Fiction PrototypeabstractThe case study presented in this paper aimed at discovering opportunities for ambient intelligence and new interaction methods for a future remote crane-operating environment. The theoretical objective was to carry out an experience-driven research project in an industrial work context, and the practical objective was to create and evaluate a future oriented science fiction prototype. The work was carried out in close co-operation with an industrial partner who was a domain expert in the field of crane industry. The aim was to focus on clearly defined user experience goals to which the industrial partner committed. The consequent immediate objective was to focus on two explicit experiences relating to 'Feeling of being motivated' and to 'Sense of community' that were placed at the core of the design investigations. These experiences were deliberated into a science fiction prototype entitled 'Remote operator's day in a future control centre'. The science fiction prototype was further introduced to users by twofold means: as a plain, simple short story and through an interactive experience path. The main contribution of the paper is to demonstrate how an emerging technology research of a new domain may employ science fiction prototyping method as a primary means of experience design investigations. Tiina Kymäläinen, Piia M. H. Perälä, Jaakko Hakulinen, Tomi Heimonen, Jobin Mathew James, Juha Pera |
Intelligent Environments | 3 |
| 2015 | Collaborative navigation in virtual worlds: how gender and game experience influence user behaviorabstractThere exists a large base of evidence for gender differences in human navigation. However, there is not much research on gender differences in collaborative aspects of navigation, including the interaction of individuals during collaborative wayfinding tasks in virtual environments. In light of this, we present a study of a collaborative virtual environment, Berlin Kompass. The goal of this study was to find out the main differences between genders in collaborative wayfinding. The application was evaluated in the context of foreign language learning in schools with over 200 students, where the users navigated through cityscapes while interacting verbally with each other. We collected and analyzed interaction logs, questionnaire data and audio and video recordings to gain insights into gender-related differences in wayfinding in virtual worlds. Our findings suggest that several differences that are evident in single user systems are not present when the collaborative aspect is added. Male users were more immersed during the task than females. One of the explaining factors for this might be video game experience. Genders also communicated differently -- males spoke in longer utterances whereas females had more, shorter utterances. Males referred more to relative directions and dynamic landmarks such as cars and pedestrians while navigating. Males with more video game experience also provided more positive subjective user experience feedback on the application. Pekka Kallioniemi, Tomi Heimonen, Markku Turunen, Jaakko Hakulinen, Tuuli Keskinen, Laura Pihkala-Posti, Jussi Okkonen, Roope Raisamo |
VRST | 4 |
| 2015 | Defining user experience goals to guide the design of industrial systemsabstractThe key prerequisite for experience-driven design is to define what experience to design for. User experience (UX) goals concretise the intended experience. Based on our own case studies from industrial environments and a literature study, we propose five different approaches to acquiring insight and inspiration for UX goal setting: Brand, Theory, Empathy, Technology, and Vision. Each approach brings in a different viewpoint, thus supporting the multidisciplinary character of UX. The Brand approach ensures that the UX goals are in line with the company's brand promise. The Theory approach utilises the available scientific knowledge of human behaviour. The Empathy approach focuses on knowing the actual users and stepping into their shoes. The Technology approach considers the new technologies that are being introduced and their positive or negative influence on UX. Finally, the Vision approach focuses on renewal, introducing new kinds of UXs. In the design of industrial systems, several stakeholders are involved and they should share common design goals. Using the different UX goal-setting approaches together brings in the viewpoints of different stakeholders, thus committing them to UX goal setting and emphasising UX as a strategic design decision. Eija Kaasinen, Virpi Roto, Jaakko Hakulinen, Tomi Heimonen, Jussi P. P. Jokinen, Hannu Karvonen, Tuuli Keskinen, Hanna Koskinen, Pertti Saariluoma, Helena Tokkonen, Markku Turunen |
Behav. Inf. Technol. | 3 |
| 2013 | Creating Immersive Audio and Lighting Based Physical Exercise Games for Schoolchildren
Jaakko Hakulinen, Markku Turunen, Tomi Heimonen, Tuuli Keskinen, Antti Sand, Janne Paavilainen, Jaana Parviainen, Sari Yrjänäinen, Frans Mäyrä, Jussi Okkonen, Roope Raisamo |
Advances in Computer Entertainment | 1 |
| 2013 | Designing Gesture-Based Control for Factory Automation
Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Jussi P. P. Jokinen, Tuuli Keskinen, Roope Raisamo |
INTERACT (2) | 2 |
| 2013 | Evaluating landmark attraction model in collaborative wayfinding in virtual learning environmentsabstractIn Virtual Learning Environments efficient navigation is a major issue, especially when it is used as a component in the learning process. This paper addresses the challenges in creating meaningful navigation routes from language learning perspective. The work is grounded on findings from a specific case on German language learning, wherein two remotely located users communicated in a wayfinding guidance scenario. The users navigated through 360-degree virtual panoramic images using body gestures and could receive communication help via spoken hints by pointing at objects in the scenery. An important design consideration is how to choose these objects, as they have both navigational importance and pedagogical significance in terms of learning the desired language. Wayfinding interactions from 21 participants were compared to the values provided by a landmark attraction model applied on the landmarks along the routes. The results show that there was a clear connection between prominence of landmarks and time spent on each panorama. This indicates that together with pedagogical planning, the model can aid in selecting the interactive content for language learning applications in virtual environments. Pekka Kallioniemi, Jaakko Hakulinen, Tuuli Keskinen, Markku Turunen, Tomi Heimonen, Laura Pihkala-Posti, Mikael Uusi-Mäkelä, Pentti Hietala, Jussi Okkonen, Roope Raisamo |
MUM | 2 |
| 2013 | Evaluating the experiential user experience of public display applications in the wildabstractStudying pervasive systems in the wild has recently gained significant interest. However, few methods exist that focus on the subjective of user experience of such systems rather than objective metrics, like performance and task success. Especially multimodal interaction in this context poses challenges to understanding how different input and output methods affect the users' experience. We present a new method for evaluating the experiential user experience of interactive systems. It combines two existing approaches from different fields: a questionnaire-based evaluation method called SUXES, intended for evaluating user expectations and experiences, and a theoretical experience framework, Experience Pyramid, originally developed for analyzing and improving experiential tourism products. The new method was used in two field studies of multimodal public display applications. Our findings show that the method is a practical approach for user experience evaluation in the wild, especially in the case of pervasive applications that aim to provide novel experiences rather than facilitate task-oriented information access. Tuuli Keskinen, Jaakko Hakulinen, Tomi Heimonen, Markku Turunen, Sumita Sharma, Toni Miettinen, Matti Luhtala |
MUM | 2 |
| 2013 | Mobile dictation for healthcare professionalsabstractWe demonstrate a mobile dictation application utilizing automatic speech recognition for healthcare professionals. Development was done in close collaboration between human-technology interaction and nursing science researchers and professionals working in the area. Our work was motivated by the need for improvements in getting spoken patient information to the next treatment steps without additional steps. In addition, we wanted to enable truly mobile spoken information entry, i.e., dictation can take place on the spot. In order to study the applicability we conducted a small-scale Wizard-of-Oz evaluation in a real hospital environment with real nurses. Our main focus was to gather subjective expectations and experiences from the actual nurses themselves. The results show true potential for our mobile dictation application and its further development. Tuuli Keskinen, Aleksi Melto, Jaakko Hakulinen, Markku Turunen, Santeri Saarinen, Tamás Pallos, Pekka Kallioniemi, Riitta Danielsson-Ojala, Sanna Salanterä |
MUM | 3 |
| 2011 | Multimodal multi-device program guide for smart conferencesabstractWe demonstrate a multimodal, multi-user, and multi-device conference program guide for conference participants. Its functionality includes access to the conference program with additional multimedia content, voting, feedback, and communication with the other participants. People can interact with the system in a multimodal way using spoken language, gestures and haptic feedback with mobile phones and shared public displays. Markku Turunen, Juho Hella, Toni Miettinen, Pellervo Valkama, Jaakko Hakulinen, Roope Raisamo |
Mobile HCI | 5 |
| 2011 | Multimodal and mobile conversational Health and Fitness Companions
Markku Turunen, Jaakko Hakulinen, Olov Ståhl, Björn Gambäck, Preben Hansen, María del Carmen Rodríguez Gancedo, Raúl Santos de la Cámara, Cameron G. Smith, Daniel Charlton, Marc Cavazza |
Comput. Speech Lang. | 2 |
| 2010 | Parallel processing of interruptions and feedback in companions affective dialogue system
Jaakko Hakulinen, Markku Turunen, Raúl Santos de la Cámara, Nigel T. Crook |
INTERSPEECH | 1 |
| 2010 | Assessment of spoken and multimodal applications: lessons learned from laboratory and field studies
Markku Turunen, Jaakko Hakulinen, Tomi Heimonen |
INTERSPEECH | 2 |
| 2010 | Visualization of Multi-sensory Meeting Information to Support AwarenessabstractAwareness of shared work activities is important for fostering professional relationships. This paper introduces visualization techniques for a meeting support system based on automatic speech recognition and multisensory meeting information capture, including voice levels and participant comments. The system has been deployed in live meetings over the past 5 months and we report meeting participants’ expectations and findings from a field study. Initial findings suggest that the proposed approach can facilitate awareness without the need for detailed meeting transcriptions and associated audio recordings, however the quality of speech recognition results and their effective visualization remain critical factors for increased adoption. Tomi Heimonen, Saila Ovaska, Markku Turunen, Jaakko Hakulinen, Juha-Pekka Rajaniemi, Kari-Jouko Räihä |
IV | 4 |
| 2010 | How was your day? An architecture for multimodal ECA systems
Raúl Santos de la Cámara, Markku Turunen, Jaakko Hakulinen, Debora Field |
SIGDIAL Conference | 3 |
| 2010 | How was your day? An Affective Companion ECA Prototype
Marc Cavazza, Raúl Santos de la Cámara, Markku Turunen, José Relaño-Gil, Jaakko Hakulinen, Nigel T. Crook, Debora Field |
SIGDIAL Conference | 5 |
| 2009 | Multimodal Media Center Interface Based on Speech, Gestures and Haptic Feedback
Markku Turunen, Jaakko Hakulinen, Juho Hella, Juha-Pekka Rajaniemi, Aleksi Melto, Erno Mäkinen, Jussi Rantala, Tomi Heimonen, Tuuli Keskinen, Hannu Soronen, Mervi Hansen, Pellervo Valkama, Toni Miettinen, Roope Raisamo |
INTERACT (2) | 2 |
| 2009 | Multimodal Interaction with Speech, Gestures and Haptic Feedback in a Media Center Application
Markku Turunen, Jaakko Hakulinen, Juho Hella, Juha-Pekka Rajaniemi, Aleksi Melto, Erno Mäkinen, Jussi Rantala, Tomi Heimonen, Tuuli Keskinen, Hannu Soronen, Mervi Hansen, Pellervo Valkama, Toni Miettinen, Roope Raisamo |
INTERACT (2) | 2 |
| 2009 | SUXES - user experience evaluation method for spoken and multimodal interactionabstractMuch work remains to be done with subjective evaluations of speech-based and multimodal systems. In particular, user experience is still hard to evaluate. SUXES is an evaluation method for collecting subjective metrics with user experiments. It captures both user expectations and user experiences, making it possible to analyze the state of the application and its interaction methods, and compare results. We present the SUXES method with examples of user experiments with different applications and modalities. Index Terms: evaluation, subjective metrics, user experience Markku Turunen, Jaakko Hakulinen, Aleksi Melto, Tomi Heimonen, Tuuli Keskinen, Juho Hella |
INTERSPEECH | 2 |
| 2009 | Speech-based and multimodal media center for different user groups
Markku Turunen, Jaakko Hakulinen, Aleksi Melto, Juho Hella, Juha-Pekka Rajaniemi, Erno Mäkinen, Jussi Rantala, Tomi Heimonen, Tuuli Keskinen, Hannu Soronen, Mervi Hansen, Pellervo Valkama, Toni Miettinen, Roope Raisamo |
INTERSPEECH | 2 |
| 2009 | User expectations and user experience with different modalities in a mobile phone controlled home entertainment systemabstractHome environment is an exciting application domain for multimodal mobile interfaces. Instead of multiple remote controls, personal mobile devices could be used to operate home entertainment systems. This paper reports a subjective evaluation of multimodal inputs and outputs for controlling a home media center using a mobile phone. A within-subject evaluation with 26 participants revealed significant differences on user expectations on and experiences with different modalities. Speech input was received extremely well, even surpassing expectations in some cases, while gestures and haptic feedback were almost failing to meet the lowest expectations. The results can be applied for designing similar multimodal applications in home environments. Markku Turunen, Aleksi Melto, Juho Hella, Tomi Heimonen, Jaakko Hakulinen, Erno Mäkinen, Tuuli Keskinen, Hannu Soronen |
Mobile HCI | 5 |
| 2008 | A comparison of input entry rates in a multimodal mobile application
Aleksi Melto, Markku Turunen, Jaakko Hakulinen, Anssi Kainulainen, Tomi Heimonen |
INTERSPEECH | 3 |
| 2008 | Voice commands in home environment - a consumer surveyabstractWe studied with telephone survey the opinions and ideas of 1004 Finns concerning domestic technology and controlling it with voice commands. There is distrust towards commanding your home environment with voice; the users doubt especially its functionality. Voice commands are regarded as unpleasant. The most positive conceived aspects of voice commands are the speed of control and the ability to free your hands for something else. Voice feedback, however, is really appreciated. Especially it is welcomed instead of alarm beeps and blinking lights. It is possible that users need first to get accustomed that a device speaks to them. Index Terms: smart home, speech interface, voice command, consumer survey Hannu Soronen, Markku Turunen, Jaakko Hakulinen |
INTERSPEECH | 3 |
| 2008 | Physically embodied conversational agents as health and fitness companionsabstractWe present a physical multimodal conversational Companion in the area of health and fitness. Conversational spoken dialogues using physical agents provide a potential interface for applications which are aimed at motivating and supporting users. Open source software called jNabServer, which enables spoken and multimodal interaction with Nabaztag/tag wireless rabbits, is presented together with other software architecture solutions applied in the development of the Companion. We also present how the Companion manages interaction with the combination of a dialogue manager and a cognitive model. Index Terms: spoken dialogue systems, multimodality, software architectures, physical agent interfaces Markku Turunen, Jaakko Hakulinen, Cameron G. Smith, Daniel Charlton, Li Zhang 0032, Marc Cavazza |
INTERSPEECH | 2 |
| 2008 | Mobidic - a mobile dictation and notetaking applicationabstractAbstract Mobile devices have become ubiquitous and reasonably po-werful and well connected. However, their physical size limits possibilities of interaction, especially document creation. Dictation in mobile setting provides one solution, but limited processing power requires that the actual speech recognition is distributed to a server computer. We present MobiDic, a distributed mobile dictation application. Its user interface solutions solve problems that arise from the technical limita-tion of mobile devices and mobile user context. Index Terms : mobile applications, dictation, audio editing 1. Introduction Dictation is one of the few real success stories of speech rec-ognition technology, and speech in general can be very effi-cient data entry method [1], although there are several chal-lenges for its use [2]. Traditional dictation applications are mostly used by specific professions, e.g., doctors and lawyers, in acoustically safe environments and on computationally adequate hardware. However, people are more and more on the move most of their day, and have immediate access only to less capable hardware, both computationally and interac-tion-wise. Furthermore, their schedule is often fragmented, so they do not have time for long-lasting dictation sessions to dictate in the traditional sense. People perceive mobile dicta-tion to be very useful, but there are also great demands for its accessibility and usability [3][4]. Thus, mobile solutions al-lowing incremental dictation and notetaking are necessary to meet the current demands. However, mobile platforms, and standard mobile phones in particular, are problematic for dictation applications be-cause of lack of processing power. There are only initial pro-totypes of embedded large-vocabulary speech recognition [5]. Most advanced dictation systems need more hardware capa-bilities than is available in current and near-future mobile phones. Because of these reasons we have focused on distri-buted solution, where the dictation and editing can be done with a mobile device but ASR takes place in a server ma-chine. Furthermore, the server stores user audio recordings and recognized texts, and maintains acoustic models and dis-tributes generated document, e.g., via email. The limited interaction capabilities offered by mobile de-vices form another fundamental issue. In particular, small screens and keypads make it hard to design efficient docu-ment editing interfaces. Most of the desktop applications are assuming standard sized screens and multimodal interaction, in particular when error correction is considered [6]. Further-more, most of them are targeted for single session, real-time interaction, not asynchronous and fragmented interaction as in mobile dictation applications. There are some commercial asynchronous mobile dicta-tion solutions available. Typically, the user dictates while on move, and the recognition takes place afterwards. In this case, the process is not interactive, and the user is not able to re-view, edit and distribute the results in the mobile settings. We present MobiDic, an asynchronous dictation and note-taking application for mobile phones. In this paper, we de-scribe the MobiDic application, its functionality and architec-ture, and then discuss the user interface solutions for incre-mental dictation, audio editing, recognition results editing, and user adaptation. Figure 1. Markku Turunen, Aleksi Melto, Anssi Kainulainen, Jaakko Hakulinen |
INTERSPEECH | 4 |
| 2008 | Integrating Planning and Dialogue in a Lifestyle Agent
Cameron G. Smith, Marc Cavazza, Daniel Charlton, Li Zhang 0032, Markku Turunen, Jaakko Hakulinen |
IVA | 6 |
| 2008 | Evaluation of predictive text and speech inputs in a multimodal mobile route guidance applicationabstractWe present initial results from an evaluation of a multimodal interface for a mobile route guidance application providing public transport information in Finland. The application includes a range of input and output modalities, such as speech input and output, a fisheye GUI, and contextual predictive text input. The application has been in public pilot use for almost a year. In this paper we present results from a formal user evaluation, focusing on predictive text input and speech input and output. Speech input outperformed other input methods, even with high error rates and slow response times. However, the domain specific predictive text input was the preferred method, surpassing users' expectations. Aleksi Melto, Markku Turunen, Anssi Kainulainen, Jaakko Hakulinen, Tomi Heimonen, Ville Antila |
Mobile HCI | 4 |
| 2007 | Design of a rich multimodal interface for mobile spoken route guidanceabstractWe present a design of a rich multimodal interface for mobile route guidance. The application provides public transport information in Finland, including support for pedestrian guidance when the user is changing between the means of transportation. The range of input and output modalities include speech synthesis, speech recognition, a fisheye GUI, haptics, contextual text input, physical browsing, physical gestures, non-speech audio, and global positioning information. Together, these modalities provide an interface that is accessible for a wide range of users including persons with various levels of visual impairment. In this paper we describe the functional aspects and the design of the interface of our publicly available prototype system. Index Terms: speech interfaces, multimodal interfaces, mobile applications, accessibility Markku Turunen, Jaakko Hakulinen, Anssi Kainulainen, Aleksi Melto, Topi Hurtig |
INTERSPEECH | 2 |
| 2006 | Evaluation of a spoken dialogue system with usability tests and long-term pilot studies: similarities and differencesabstractWe present findings from the long-term study of a speech-based bus timetable system. After the deployment of the prototype system we have collected data from real usage for 30 months. In addition, we have conducted usability tests to get subjective ratings of the pilot system. The comparison of these evaluations shows that the results obtained with usability tests differ significantly from those gained from the real usage, and the data of the initial use differs significantly from the data collected after that. For example, the differences in help requests, interruptions, speech recognition rejections, silence timeouts, and repeat requests are highly significant, and in some cases, such as explicit quit requests, enormous (65 % versus 3%). Index Terms: spoken dialog systems, evaluation, pilot studies 1. Markku Turunen, Jaakko Hakulinen, Anssi Kainulainen |
INTERSPEECH | 2 |
| 2005 | Visualization of spoken dialogue systems for demonstration, debugging and tutoringabstractGraphical elements have been found very useful when spoken dialogue systems are developed and demonstrated. However, most of the spoken dialogue systems are designed for speechonly interaction and are very hard to extent to contain graphical elements. We introduce a general model to visualize speech interfaces. Based on the model we present an implemented visualization framework, and several example visualizations for demonstrations, debugging, and interactive tutoring of speech applications. 1. Jaakko Hakulinen, Markku Turunen, Esa-Pekka Salonen |
INTERSPEECH | 1 |
| 2005 | Towards generic spatial object model and route guidance grammar for speech-based systems
Perttu Prusi, Anssi Kainulainen, Jaakko Hakulinen, Markku Turunen, Esa-Pekka Salonen, Leena Helin |
INTERSPEECH | 3 |
| 2005 | Distributed dialogue management for smart terminal devicesabstractMobile devices, such as smartphones and personal digital assistants, can be used to implement efficient speech-based and multimodal interfaces. Most of the systems are serverbased, and there is a need to distribute the dialogue management tasks between the terminal devices and the server. Since the technologies are not mature and the platforms are constantly changing, approaches that support generic and reusable components are needed. We describe a model where the tasks of the dialogue management are divided into two parts. The server handles the overall coordination of the dialogue by generating dialogue task descriptions. Smart terminal devices realize the low-level decisions by executing individual dialogue tasks. A prototype of multimodal dialogue application using the model is presented. It has two different user interface realizations using standard telephony equipment and smartphones. 1. Esa-Pekka Salonen, Markku Turunen, Jaakko Hakulinen, Leena Helin, Perttu Prusi, Anssi Kainulainen |
INTERSPEECH | 3 |
| 2004 | Tutor design for speech-based interfacesabstractSpeech-based applications commonly come with web-based or printed manuals. Alternatively, the dialogue can be designed so that users should be able to start using the application on their own. We studied an alternative approach, an integrated tutor. The tutor participates in the interaction when new users learn to use a speech-based system. It teaches the users how to operate the system and monitors user actions to be certain that the users do indeed learn. In this paper we describe our experiences with the design and the iterative development of an integrated tutor. Expert evaluation and two user tests were conducted with different versions of the tutor. The results show that the tutor can effectively guide new users. We identify the six most important lessons learned, the most important being that it is essential to spot problems by monitoring user actions, especially when novice users are tutored. Jaakko Hakulinen, Markku Turunen, Esa-Pekka Salonen, Kari-Jouko Räihä |
Conference on Designing Interactive Systems | 1 |
| 2004 | Flexible dialogue management using distributed and dynamic dialogue control
Mikko Hartikainen, Markku Turunen, Jaakko Hakulinen, Esa-Pekka Salonen, J. Adam Funk |
INTERSPEECH | 3 |
| 2004 | Robust and adaptive architecture for multilingual spoken dialogue systemsabstractWe present how robustness and adaptivity can be supported by the spoken dialogue system architecture. AthosMail is a multilingual spoken dialogue system for e-mail domain. It is being developed in the EU-funded DUMAS project. It has flexible system architecture supporting multiple components for input interpretation, dialogue management and output generation. In addition to language differences, these components have great variation in their approaches for spoken interaction. For example, the dialogue management components implement different dialogue control models. The system architecture makes the interaction robust and adaptive by utilizing different approaches for spoken interaction in a single application by selecting suitable components at runtime. 1. Markku Turunen, Esa-Pekka Salonen, Mikko Hartikainen, Jaakko Hakulinen |
INTERSPEECH | 4 |
| 2003 | Agents for integrated tutoring in spoken dialogue systemsabstractIn this paper, we introduce the concept of integrated tutoring in speech applications. An integrated tutoring system teaches the use of a system to a user while he/she is using the system in a typical manner. Furthermore, we introduce the general principles of how to implement applications with integrated tutoring agents and present an example implementation for an existing e-mail system. The main innovation of the approach is that the tutoring agents are part of the application, but implemented in a way which makes it possible to plug them into the system without modifying it. This is possible due to a set of small, stateless agents and a shared Information Storage provided by our system architecture. Integrated tutoring agents are easily expandable and configurable, and general agents can be shared between applications. We have also received positive feedback about integrated tutoring in initial user tests conducted with the implementation. Jaakko Hakulinen, Markku Turunen, Esa-Pekka Salonen |
INTERSPEECH | 1 |
| 2003 | Jaspis^2 - an architecture for supporting distributed spoken dialoguesabstractIn this paper, we introduce an architecture for a new generation of speech applications. The presented architecture is based on our previous work with multilingual speech applications and extends it by introducing support for synchronized distributed dialogues, which is needed in emerging application areas, such as mobile and ubiquitous computing. The architecture supports coordinated distribution of dialogues, concurrent dialogues, system level adaptation and shared system context. The overall idea is to use interaction agents to distribute dialogues, use an evaluation mechanism to make them dynamically adaptive and synchronize them by using a coordination mechanism with triggers and transactions. We present experiences from several applications written on top of the freely available architecture. Markku Turunen, Jaakko Hakulinen |
INTERSPEECH | 2 |
| 2001 | Agent-based error handling in spoken dialogue systems
Markku Turunen, Jaakko Hakulinen |
INTERSPEECH | 2 |
| 2000 | Jaspis - a framework for multilingual adaptive speech applications
Markku Turunen, Jaakko Hakulinen |
INTERSPEECH | 2 |