Ann Li

Hand-written name, "Ann Li"

Ann Li

is a multidisciplinary designer and researcher exploring experiential narratives and connective systems

Best viewed on desktop

Ann Li

Neurobiologist-turned designer driven by sense-making, systems thinking, and storytelling. Seeking full-time interaction, experience, or product design roles.

Work spans the digital and physical, including functional user interfaces, immersive experiences, research, and creative strategy.

Augmented Learning Tools

Designing student-driven tools to train skilled workers for future opportunities in creative arts and manufacturing

Roles

Design Research
Interaction Design
Product Strategy
Interface Design
Workshop Facilitation
XR Prototyping

Context

Jan 2023-May 2024

Team

Zhenfang Chen
Andrew Knowles
Tate Johnson
Semina Yi
Yumeng Zhuang
Dina El-Zanfaly
Daragh Byrne

Overview
Outcomes
ResearchApproachDesignReflections

Overview

Despite the prevalence of welding in construction, manufacturing, and creative arts, training novice welders is a time-intensive and challenging process.

  1. Metal welding is a craft manufacturing skill that can be unusually difficult to externalize and represent to novices, with most current training methods using non-immersive simulations for skill acquisition.
  2. Welding practice and training is resource intensive, consuming materials, energy, and time.
  3. Building competency requires novices to build deep embodied knowledge, such as hand-eye coordination, muscle memory, and sound identification, and adjust to overwhelming sensory stimuli.

Hands-on skills and training are pressing needs for growing the American workforce. It has become increasingly challenging to attract and retain novice welders.

To explore these challenges, a series of co-design workshops were conducted with a youth program in welding and fabrication. Opportunities for mixed reality, sensing, and tinyML processes to augment welding training and practice were identified, leading to the design and deployment of an extended reality welding helmet and torch.

Contributions

I led the design and execution of various research plans, including the facilitation of a series of co-design workshops, participatory ideation sessions, and evaluative research. I also contributed to feature development, UI design, and overall project approach.

Outcomes

XR Weld is a mixed reality system that provides real-time, in-helmet, and in-situ feedback to augment live welding training.

An XR system assists with limitations of traditional welding education. By enabling users to encounter real behaviors, forces, sounds, and overall experience of an active weld, we provide a novel in-situ experience with spatial and biometric analysis.

The designed system allows us to explore other aspects of embodied learning in welding practice, including the analysis of biometric data, performance analysis, and the inclusion of meditation to further augment the educational experience.

Modified welding helmet and gun prototype
In-helmet UI feedback

We apply machine learning to a lightly-modified off-the-shelf XR and welding setup to enable in-situ welding training, enhancing the embodied learning of welding in three key ways.

  1. Visual XR Guides and Integrated Motion Sensing. Combined motion-sensing and visual XR feedback helps improve proprioceptive and embodied learning.
  2. Sensing Sonic Cues During Welding Practice. Acoustic sensing focuses learner attention on non-visual cues of weld performance.
  3. Mediated Meditation and Regulation. Biometric sensing enhances mindfulness and stress management in sensorially challenging environments.
Simplified system diagram

Research

We worked closely with the Industrial Arts Workshop (IAW), a non-profit youth welding training program, to inform our approach which was grounded in co- and participatory design.

Community Workshops

A series of three on-site community workshops and focus groups were conducted to explore welding instructor and student needs, achieve working understanding of learning contexts, and identify design opportunities along key moments of the curriculum.

Focus groups with instructors and leadership

Facilitating immersive, activity-based engagement

Participating in welding training immersion

We worked with stakeholders (instructors and students) in their context of use, making several visits to the IAW site, where we experienced firsthand the challenges, highlights, goals, and considerations encountered by novice welders.

It was important to us that we emphasize a participatory design approach, rather than classical theory-based or researcher-led approaches. In order to accomplish this, I aimed to identify and provide suitable tools and processes for our community partners. I developed workshop agendas that incorporated various methods, including projective techniques (diagramming, journey mapping, storyboarding), and creative activities.

Experiential co-design workshops conducted with community collaborators at the partner site

Designing new methods that gain creative input and trust

However, IAW partners were not familiar with XR interfaces, and tools and techniques for co-designing mixed reality experiences are limited in adaptability and transferability. As a result, we opted to focus on storyboarding and scenario making as an accessible means to co-design. This also resulted in the design of novel creative toolkits for participatory prototyping of XR interfaces, an accessible, flexible, and intuitive medium to solicit design input and feedback on features.

Participatory co-design of HUD interface
Accessible prototyping for XR visual interfaces

Contributions

The novelty of our work includes the ability for it to be used as an in-situ, functioning tool, as well as our holistic approach to augmentation.

Prior work in the areas of extended reality and welding create isolated, fully immersive simulations of welding, rather than an augmentation of actual welding in context. Existing products and research include full VR welding experiences (training without a functional helmet or torch, but a visual augmentation with hand tracking) and AR/computer vision-based simulations using tracking markers.

Our system is implemented to work with a live spark, in real welding contexts. Beyond training muscle memory, our system also uses additional sensors and the Quest's onboard sensors to promote good practice through feedback on mindful breathing, weld performance, and other attributes.

Device assemblies
Updated system

‍

Impact

We developed, tested, and deployed a mixed-reality set-up that provides interactive instructions and feedback to trainees, helps them achieve and explore creative structures and outcomes, and engages them in learning and retention within the training program.

We partnered with the Industrial Arts Workshop (IAW), a non-profit which focuses on training teenagers in the underserved community of Hazelwood through creative arts welding projects. This project will enable us in the future to democratize learning to transfer hands-on skills by creating immersive interfaces in many fields such as operating machines, woodworking, and physical computing, and to improve recruitment and retention of a skilled workforce. There is a need for such a training set-up for all ages and vocations.

‍

Publications and Press

Details on our co-design process and hardware development, system setup, training specifics, related works, and citations

Augmenting Embodied Learning in Welding Training: The Co-Design of an XR- and tinyML-Enabled Welding System for Creative Arts and Manufacturing Training
‍ACM Tangible Embedded and Embodied Interaction 2024
Honorable Mention

Augmenting Welding Training: An XR Platform to Foster Muscle Memory and Mindfulness for Skills Development
‍ACM Interactive Surfaces and Spaces 2023
Best Demo

XRweld: An In-Situ Extended Reality Platform for Welding Education
ACM SIGGRAPH 2024

Machine learning and extended reality used to train welders
Feature: Carnegie Mellon University College of Engineering

Build Back Better Grant

‍

Click to navigate to the top
A Changing Arctic, Carnegie Museum of Natural History
Augmented Learning Tools
Cara, CVS Pharmacy
Wishwell
Artwork
Music Programming
Hand-written name, "Ann Li"
InstagramTwitterLinkedinBehanceDribbble

Ann Li 2020

Don't be a stranger
If you think I'd be a good fit for your project or team, let's connect.
© 2024 Ann Li
LinkedIn link