Virtual reality has the potential to simulate a variety of real-world scenarios for training- and entertainment-purposes, as it has the ability to induce a sense of 0́−presence0́+: the illusion that the user is physically transported to another location and is really 0́−there0́+. VR and VR-technologies have seen a recent market resurgence due to the arrival of affordable, mass-market VR-display systems, such as the Oculus Rift, HTC Vive, PlayStation VR, Samsung GearVR, and Google Cardboard. However, the use of tactile feedback to convey information about the virtual environment is often lacking in VR applications. This study addresses this lack by proposing the use of binaural audio in VR to induce illusory tactile feedback. This is done by examining the literature on intersensory illusions as well as the relationship between audio and tactile feedback to inform the design of a software prototype that is able to induce the desired feedback. This prototype is used to test the viability of such an approach to induce illusory tactile feedback and to investigate the nature of this feedback. The software prototype is used to collect data from users regarding their experiences of this type of feedback and its underlying causes. Data collection is done through observation, questionnaires, interviews, and focus groups and the results indicate that the use of binaural audio in VR can be used to effectively induce an illusory sense of tactile feedback in the absence of real-world feedback. This study contributes insights regarding the nature of illusory sensations in VR, focusing on touch-sensations. This study also provides consolidated definitions of immersion and presence as well as a consolidated list of aspects of immersion, both of which are used to detail the relationship between immersion, presence, and illusory tactile feedback. Findings provide insight into the relationship between the design of audio in VR and its ability to alter perception in the tactile modality. Findings also provide insight into aspects of VR, such as presence and believability, and their relationship to perception across various sensory modalities.
This seven-volume set LNCS 14054-14060 constitutes the proceedings of the 25th International Conference, HCI International 2023, in Copenhagen, Denmark, in July 2023. For the HCCII 2023 proceedings, a total of 1578 papers and 396 posters was carefully reviewed and selected from 7472 submissions. Additionally, 267 papers and 133 posters are included in the volumes of the proceedings published after the conference, as “Late Breaking Work”. These papers were organized in the following topical sections: HCI Design and User Experience; Cognitive Engineering and Augmented Cognition; Cultural Issues in Design; Technologies for the Aging Population; Accessibility and Design for All; Designing for Health and Wellbeing; Information Design, Visualization, Decision-making and Collaboration; Social Media, Creative Industries and Cultural Digital Experiences; Digital Human Modeling, Ergonomics and Safety; HCI in Automated Vehicles and Intelligent Transportation; Sustainable Green Smart Cities and Smart Industry; eXtended Reality Interactions; Gaming and Gamification Experiences; Interacting with Artificial Intelligence; Security, Privacy, Trust and Ethics; Learning Technologies and Learning Experiences; eCommerce, Digital Marketing and eFinance.
Despite recent advances in technology, current Virtual Reality (VR) experiences have important limitations, including users' inability to walk around in large virtual environments or receive realistic haptic feedback. In this dissertation, I argue that we do not need to strive for perfect VR hardware--hardware that can closely replicate users' real-world experiences--to address these challenges. Rather, I argue that we can take advantage of the users' perceptual limits and motor control adaptability to improve the user experience instead. VR provides a unique opportunity to programmatically overwrite users' sensory signals and transform how movements are visually rendered. We can leverage this property and tap into the human sensorimotor control loop to not only overcome the current limitations of VR technology, but also to extend our abilities beyond what we can do in the real world. In part I of this dissertation, I explore illusory interactions that apply subtle remappings to users' movements in VR. While users do not notice the resulting sensory discrepancy, they alter their movements in response. I demonstrate how we can intentionally use these unnoticeable changes in movements to improve the perceived performance of encountered-type haptic devices, such as drones and shape displays. In part II, I highlight that we can take these remappings one step further and apply noticeable transformations, which I refer to as ``beyond-real.'' I present a conceptual framework for describing these interactions and conduct a survey to categorize the types of transformations explored in prior research. I then demonstrate that these interactions can be utilized to address the challenge of locomotion in VR. While users likely have not experienced such transformations in the real world before, they can learn to adapt to the novel dynamics and remain in control of their movements. Throughout the dissertation, I demonstrate that by understanding the limits of human sensory integration and motor control, we can design effective VR interactions that create an illusion of improved haptics and extend users' abilities beyond what is possible in the real world.
The use of non-intrusive virtual environments is gaining more and more importance but was focused mainly on addressing the visual sense. However, the human perception consists not only of visual input and thus it would be worthwhile to create multi-modal and interactive virtual environments. This thesis describes the techniques required to include the acoustic component into a virtual environment and furthermore the implementation of a software system, which takes advantage of these techniques to create complex acoustical scenes in real time. The system is based on the binaural technology. It features spatially distributed sound sources which are utilized to create an environment that is as authentic as possible. This comprises a description of the source, including its relevant angle-, distance- and time- dependent radiation, the sound distribution in the virtual scene (room acoustics), the perception-related consideration of all sound field components, as well as the exact reproduction of the artificial sound at the ears of the user. The focus of this thesis is put on the reproduction technology. In this context, an approach for dynamic crosstalk cancellation is presented, which enables a loudspeaker-based reproduction. The required filters are processed in real time on the basis of the position data and measured transfer functions of the outer ear. Furthermore the integration of this spatial audio system into a five-sided Virtual Reality display system is described and evaluated.
This paper presents a set of experiments in which a human user feels haptic sensations. These sensations are in fact haptic illusions generated by a visual effect. Then these haptic illusions are described and analysed. These haptic illusions were generated by the use of a pseudo-haptic feedback system. It is a system combining an isometric input device and visual feedback. The experimental apparatus did not use any force feedback interlace The paper addresses the role of action in the perception loop subjects felt a reactive force corresponding to their own sensory-motor command. In addition subjects had to "participate" in the illusion process by choosing the cognitive strategy which led to the illusion. In the future, the use of the concept of illusion might improve or simplify VR simulations and pave the way to a better understanding of human perception.
"Sense of Touch and its Rendering" presents a unique and interdisciplinary approach highlighting the field of haptic research from a neuropsychological as well as a technological point of view. This edited book is the outcome of the TOUCH-HapSys European research project and provides an important contribution towards a new generation of high-fidelity haptic display technologies. The book is structured in two parts: A. Fundamental Psychophysical and Neuropsychological Research and B. Technology and Applications. The two parts are not however separated, and the many connections and synergies between the two complementary domains of research are highlighted in the text. The eleven chapters discuss the recent advances in the study of human haptic (kinaesthetic, tactile, temperature) and multimodal (visual, auditory, haptic) perception mechanisms. Besides the theoretical advancement, the contributions survey the state of the art in the field, report a number of practical applications to real systems, and discuss possible future developments.
While virtual reality (VR) has influenced fields as varied as gaming, archaeology and the visual arts, some of its most promising applications come from the health sector. Particularly encouraging are the many uses of VR in supporting the recovery of motor skills following accident or illness. Virtual Reality for Physical and Motor Rehabilitation reviews two decades of progress and anticipates advances to come. It offers current research on the capacity of VR to evaluate, address, and reduce motor skill limitations and the use of VR to support motor and sensorimotor function, from the most basic to the most sophisticated skill levels. Expert scientists and clinicians explain how the brain organizes motor behavior, relate therapeutic objectives to client goals and differentiate among VR platforms in engaging the production of movement and balance. On the practical side, contributors demonstrate that VR complements existing therapies across various conditions such as neurodegenerative diseases, traumatic brain injury and stroke. Included among the topics: Neuroplasticity and virtual reality. Vision and perception in virtual reality. Sensorimotor recalibration in virtual environments. Rehabilitative applications using VR for residual impairments following stroke. VR reveals mechanisms of balance and locomotor impairments. Applications of VR technologies for childhood disabilities. A resource of great immediate and future utility, Virtual Reality for Physical and Motor Rehabilitation distills a dynamic field to aid the work of neuropsychologists, rehabilitation specialists (including physical, speech, vocational and occupational therapists), and neurologists.
In this text, philosophers, psychologists and art historians explore the implications of theories of vision for our understanding of the nature of pictorial representation and picture perception.
Sound, devoid of meaning, would not matter to us. It is the information sound conveys that helps the brain to understand its environment. Sound and its underlying meaning are always associated with time and space. There is no sound without spatial properties, and the brain always organizes this information within a temporal–spatial framework. This book is devoted to understanding the importance of meaning for spatial and related further aspects of hearing, including cross-modal inference. People, when exposed to acoustic stimuli, do not react directly to what they hear but rather to what they hear means to them. This semiotic maxim may not always apply, for instance, when the reactions are reflexive. But, where it does apply, it poses a major challenge to the builders of models of the auditory system. Take, for example, an auditory model that is meant to be implemented on a robotic agent for autonomous search-&-rescue actions. Or think of a system that can perform judgments on the sound quality of multimedia-reproduction systems. It becomes immediately clear that such a system needs • Cognitive capabilities, including substantial inherent knowledge • The ability to integrate information across different sensory modalities To realize these functions, the auditory system provides a pair of sensory organs, the two ears, and the means to perform adequate preprocessing of the signals provided by the ears. This is realized in the subcortical parts of the auditory system. In the title of a prior book, the term Binaural Listening is used to indicate a focus on sub-cortical functions. Psychoacoustics and auditory signal processing contribute substantially to this area. The preprocessed signals are then forwarded to the cortical parts of the auditory system where, among other things, recognition, classification, localization, scene analysis, assignment of meaning, quality assessment, and action planning take place. Also, information from different sensory modalities is integrated at this level. Between sub-cortical and cortical regions of the auditory system, numerous feedback loops exist that ultimately support the high complexity and plasticity of the auditory system. The current book concentrates on these cognitive functions. Instead of processing signals, processing symbols is now the predominant modeling task. Substantial contributions to the field draw upon the knowledge acquired by cognitive psychology. The keyword Binaural Understanding in the book title characterizes this shift. Both books, The Technology of Binaural Listening and the current one, have been stimulated and supported by AABBA, an open research group devoted to the development and application of models of binaural hearing. The current book is dedicated to technologies that help explain, facilitate, apply, and support various aspects of binaural understanding. It is organized into five parts, each containing three to six chapters in order to provide a comprehensive overview of this emerging area. Each chapter was thoroughly reviewed by at least two anonymous, external experts. The first part deals with the psychophysical and physiological effects of Forming and Interpreting Aural Objects as well as the underlying models. The fundamental concepts of reflexive and reflective auditory feedback are introduced. Mechanisms of binaural attention and attention switching are covered—as well as how auditory Gestalt rules facilitate binaural understanding. A general blackboard architecture is introduced as an example of how machines can learn to form and interpret aural objects to simulate human cognitive listening. The second part, Configuring and Understanding Aural Space, focuses on the human understanding of complex three-dimensional environments—covering the psychological and biological fundamentals of auditory space formation. This part further addresses the human mechanisms used to process information and interact in complex reverberant environments, such as concert halls and forests, and additionally examines how the auditory system can learn to understand and adapt to these environments. The third part is dedicated to Processing Cross-Modal Inference and highlights the fundamental human mechanisms used to integrate auditory cues with cues from other modalities to localize and form perceptual objects. This part also provides a general framework for understanding how complex multimodal scenes can be simulated and rendered. The fourth part, Evaluating Aural-scene Quality and Speech Understanding, focuses on the object-forming aspects of binaural listening and understanding. It addresses cognitive mechanisms involved in both the understanding of speech and the processing of nonverbal information such as Sound Quality and Quality-of- Experience. The aesthetic judgment of rooms is also discussed in this context. Models that simulate underlying human processes and performance are covered in addition to techniques for rendering virtual environments that can then be used to test these models. The fifth part deals with the Application of Cognitive Mechanisms to Audio Technology. It highlights how cognitive mechanisms can be utilized to create spatial auditory illusions using binaural and other 3D-audio technologies. Further, it covers how cognitive binaural technologies can be applied to improve human performance in auditory displays and to develop new auditory technologies for interactive robots. The book concludes with the application of cognitive binaural technologies to the next generation of hearing aids.
