Recent advances in VR and AR technology have enabled interactive graphics applications to support healthcare professionals in training, diagnosis, planning, and treatment. This field has progressed enough to warrant a course that can inspire new ideas within the graphics community. Medical images create virtual human anatomy models, allowing for natural interaction and visualization in healthcare scenarios. VR and AR are conceptually different and suited for different types of problems. VR is immersive and suitable for learning anatomy, surgical skills, and analyzing 3D medical data. AR, on the other hand, overlays helpful information onto the physical environment, making it useful for tasks such as communication with patients and training assistants. However, several challenges, such as nonstandard equipment and disorientation, still limit the widespread use of these technologies. This course will cover current advances and challenges in this area, including integrating AI techniques.

Joaquim Jorge holds the UNESCO Chair of Artificial Intelligence & Extended Reality at the University of Lisboa, Portugal. He joined Eurographics in 1986 and ACM/SIGGRAPH in 1989. He is Editor-in-Chief of the Computers and Graphics Journal, Eurographics Fellow, ACM Distinguished Member, and member of IEEE Computer Society Board of Governors. He organized 50+ conferences, including Eurographics 2016 (IPC CO-Chair), IEEE VR 2020/21/22 as co-(papers)chair, and ACM IUI 2012 (IPC co-chair). He served on 210+ program committees and (co)authored over 360 peer-reviewed publications and five books. His research interests include graphics, virtual reality, and advanced HCI techniques applied to health technologies.

In many fields, procedures define work and activities. These procedures consist of well-defined sequences of instructions that are followed in a specific order and according to specific rules. For example, maintenance work in industry is defined by procedures that ensure that work is performed according to maintenance strategy, policies, and programs. However, these procedures can be complex and difficult to remember, requiring learning and training for new employees and refresher training for existing personnel. This is also true in other areas, such as learning safety and medical procedures, training employees, and improving decision-making processes. Procedures are used in almost everything we do, such as reading, writing, calculating, driving, cooking, and classifying concepts.
To teach procedural skills well, teachers need to provide activity-specific instructions that describe the logical connections between activities...according to the form of the procedure. Extended reality technologies (XR) can be used to create immersive and interactive learning environments that help learners acquire procedural knowledge and skills through self-training and self-assessment. XR technologies can simulate realistic environments that help learners practice procedures in a safe and controlled environment. For example, a trainee can practice complex medical procedures in a virtual environment before performing them on real patients. In addition, XR can enhance the learning experience by providing multimedia, interactive information, context awareness, and adaptive information delivery. Learners can interact with virtual objects, receive audio and visual feedback, and track their performance and progress in real time, allowing them to identify and correct errors.
The integration of artificial intelligence (AI) systems can significantly improve self-learning systems by making them more adaptable and tailored to learners' needs. In particular, AI enables the implementation of adaptive learning approaches that constantly monitor and analyze learners' performance and adapt content and learning methods to their needs and progress. Such adaptive learning approaches can be supported by a virtual instructor with conversational capabilities. This instructor has two main functions: providing vocal and visual instructions to deliver learning content, and answering to direct requests from learners with explanations and clarifications. This interaction is critical to maintaining motivation and enhancing the learning experience. Finally, AI can be used to control realistic NPCs that actively participate in complex learning scenarios.
This course on procedure learning in XR will discuss best practices for creating effective XR/AI-based learning programs and experiences that meet the needs of diverse learners and achieve specific learning objectives.

Andrea Bottino is Associate Professor at the Department of Control and Computer Engineering of the Politecnico di Torino, where he heads the Computer Graphics and Vision research group of the same university. His current research interests include Computer Vision, Machine Learning, Human Computer Interaction, Serious Games, and Virtual and Augmented Reality.

As we see augmented reality (AR) applications move from research labs to commercial applications, the need for usable AR-based systems has become more and more evident. Despite the fact that AR technology fundamentally changes the way we visualize, use, and interact with computer-based information, we are only just recently starting to see real investment in the design and development of user interface (UI) designs that truly leverage AR’s promise. While traditional UX methods can be applied to determine what information should be presented to users, these approaches do not tell us how and when information should be presented to the user. Given the infancy of the AR UI design field, we mostly see smartphone, tablet and even desktop UI metaphors migrating to AR, in what we can consider first-gen AR user interfaces.
With recent advancements in computer vision and machine learning, we have a unique opportunity to consider intelligent next-generation AR user interfaces. In industrial settings for example, we can consider these next-gen AR interfaces to adapt to, the worker, their work progress and quality, as well as the overall work environment. In this talk, I discuss the implications for AI for next-generation AR user interfaces. Specifically, I provide some specific examples of how AI-powered AR user interfaces could adapt to users’ contexts, and touch on both the promise and challenges associated with creating and fielding these next-gen UIs. I conclude with open questions regarding issues of inclusion, privacy and agency.

Dr. Joseph L. Gabbard is an Associate Professor of Human Factors at Virginia Tech. He holds Ph.D., M.S. and B.S. degrees in Computer Science (HCI) and a B.A. in Sociology from Virginia Tech. Dr. Gabbard’s work centers on human-computer interaction in augmented and virtual reality. For nearly 25 years, Gabbard has been researching new methods of design and evaluation for interactive virtual and augmented reality systems. Currently, Dr. Gabbard directs the COGnitive Engineering for Novel Technologies (COGENT) Lab, conducting basic and applied HCI and human factors research. His work focuses on the application of principles and theories from several disciplines to the design of augmented reality user interfaces, information presentation and user interaction. His research has been funded by the National Science Foundation, National Institute of Standards and Technology, Microsoft, Meta, the National Institutes of Health, the Office of Naval Research, and several automotive manufacturers.

This lesson introduces hand tracking and gesture recognition using neural networks, implemented with Keras and TensorFlow, and integrated into the Unity 3D game engine. Through practical exercises, students will learn the principles of neural network design and training and the implementation of hand tracking and gesture recognition in Unity 3D. By the end of the lesson, students will have a fundamental understanding of the potential of these technologies in interactive systems and the skills to implement them using Keras, TensorFlow, and Unity 3D.

Nicola Capece is a postdoctoral researcher at the University of Basilicata in Italy. His research interests encompass Real-Time and Offline Rendering, Deep Learning, Computer Graphics, eXtended Reality (XR), and Human-Computer Interaction (HCI). He has authored numerous papers published in international journals and conferences. He is involved in several research projects concerning XR and HCI for cultural heritage storytelling, virtual dressing rooms, virtual reality 3D modelling, and hand gesture recognition systems. Additionally, he is an adjunct professor of Software Engineering and Virtual Archeology Lab courses.

Casa delle Tecnologie Emergenti

Casa per Ferie Sant’Anna