Ann Li

Hand-written name, "Ann Li"

Ann Li

is a multidisciplinary designer and researcher crafting creative research-driven experiences across platforms

Best viewed on desktop

Ann Li

Neurobiologist-turned designer driven by sense-making, systems thinking, and storytelling. Open to full-time product, interaction, experience design roles.

Work spans the digital and physical, including functional user interfaces, immersive experiences, mixed-methods research, and creative strategy.

Augmented Learning Tools

Designing tools to train skilled workers for future opportunities in creative arts and manufacturing

Roles

Interaction Design
UX Research
Product Strategy
Interface Design
Workshop Facilitation
XR Prototyping

Context

Jan 2023-May 2024

Team

Zhenfang Chen
Andrew Knowles
Tate Johnson
Semina Yi
Yumeng Zhuang

Advised by Dina El-Zanfaly,
Daragh Byrne

Tools

Figma
Bezi
Unity

‍

Overview
Outcomes
ResearchApproachDesignReflections

Overview

Hands-on skills and training are pressing needs for growing the American workforce, yet it has become increasingly difficult to attract and retain novice welders.

Potential extensions of this system: outreach programs and recruitment demos provide an engaging, accessible, and immersive introduction to welding

Challenge

Despite the prevalence of welding in construction, manufacturing, and creative arts, training novice welders is a time-intensive and challenging process.

  1. Metal welding is a skill that can be unusually difficult to externalize and represent to novices, with most training methods using non-immersive simulations for skill acquisition.
  2. Welding practice and training is resource intensive, consuming materials, energy, and time.
  3. Building competency requires beginners to build deep embodied knowledge, such as hand-eye coordination, muscle memory, and sound identification, and adjust to overwhelming sensory stimuli.

To explore these challenges, a series of co-design workshops were conducted with a youth program in welding and fabrication. Opportunities for mixed reality, sensing, and tinyML processes to augment welding training and practice were identified, leading to the design and deployment of an extended reality welding helmet and torch.

Contributions

I led the scoping and design of various research studies, including the facilitation of a series of co-design workshops, participatory ideation sessions, and evaluative research studies. I also contributed to feature development, spatial UI design, and overall product approach.

Outcomes

XRweld is a mixed reality system that provides real-time, in-helmet, and contextual feedback to augment live welding training.

The designed system allows us to explore other aspects of embodied learning in welding practice, including the analysis of biometric data, performance analysis, and the inclusion of meditation to further augment the educational experience.

System Walkthrough

Initialization

The student welder adjusts the Quest optics, headband fit settings, and ensures that the proper welding safety equipment is worn.

Calibration

Using the gun to place coordinate locations for the start and end of the weld, the welder links the real world weld line to a graphic representation in the XR display.

Pre-weld feedback and meditation

Before starting, the welder can use the breath-controlled meditation program and gun position monitoring to focus and prepare for a successful weld.

‍

Active weld feedback

During welds the display dims automatically, allowing the welder to see the molten weld bead along with XR display elements, which provide responsive feedback on the welder’s performance.

Reflection and evaluation

Automatically generated 3D weld lines can be reviewed post-weld from inside the headset, allowing the welder to monitor their variance from an ideal path and reflect on overall performance.

Realtime instructor view

At any time, an instructor or student can review realtime or recorded point of view (POV) footage of the experience to monitor behaviors or analyze difficult scenarios

An XR system assists with limitations of traditional welding education

By enabling users to encounter real behaviors, forces, sounds, and overall experience of an active weld, we provide a novel in-situ experience with spatial and biometric analysis.

Custom UI delivers real-time prompts, responsive feedback, and summarizes performance metrics for individual welds
Modified welding helmet and gun prototype

Research

We worked closely with the Industrial Arts Workshop (IAW), a non-profit youth welding training program, to inform our approach which was grounded in co- and participatory design.

Community Workshops

A series of three on-site community workshops and focus groups were conducted to explore welding instructor and student needs, achieve working understanding of learning contexts, and identify design opportunities along key moments of the curriculum.

Focus groups with instructors and leadership

Facilitating immersive, activity-based engagement

Participating in welding training immersion

We worked with stakeholders (instructors and students) in their context of use, making several visits to the IAW site, where we experienced firsthand the challenges, highlights, goals, and considerations encountered by novice welders.

It was important to us that we emphasize a participatory design approach, rather than classical theory-based or researcher-led approaches. In order to accomplish this, I aimed to identify and provide suitable tools and processes for our community partners. I developed workshop agendas that incorporated various methods, including projective techniques (diagramming, journey mapping, storyboarding), and creative activities.

Experiential co-design workshops conducted with community collaborators at the partner site

Designing new methods that gain creative input and trust

However, IAW partners were not familiar with XR interfaces, and tools and techniques for co-designing mixed reality experiences are limited in adaptability and transferability. As a result, we opted to focus on storyboarding and scenario making as an accessible means to co-design. This also resulted in the design of novel creative toolkits for participatory prototyping of XR interfaces, an accessible, flexible, and intuitive medium to solicit design input and feedback on features.

Participatory co-design of HUD interface
Accessible prototyping for XR visual interfaces

‍

‍

‍

‍

‍

‍

‍

‍

‍

Feedback and Refinement

Designing and testing for understandable, unobstrusive user interfaces

The basic user interface was co-designed (see above), utilizing transparent acrylic sheets in front of welding work areas to allow participants to annotate and organize functional areas within their field of view.

Persistent "Dashboard" UI

Separate 'pods’ of performance data are arranged peripherally above the viewport, maintaining a direct line-of-site in the center of the field of view (FOV). This layout was then transferred into Unity, with key metrics such as travel speed, travel angle, work angle, and standoff distance comprising the primary data ‘clusters’ in the viewport.

Persistent UI

After introducing this UI to workshop participants and conducting preliminary user testing, we discovered that the persistent UI at the top of users' FOV was distracting when trying to maintain a consistent posture and movement.

Contextual "Attached" UI

Based on rounds of user feedback, the visual UI evolved to be more contextually relevant, progressively disclosing elements when necessary and reducing cognitive load by hiding information. This glanceable UI reflects live feedback, with gauges and metrics appearing only when needed (for example, when a user drifts too far above or below a preset acceptable range).

Updated contextual UI with toggle-able indicators
4
needs
12
themes
36
subtopics
12
opportunities

We apply machine learning to a lightly-modified off-the-shelf XR and welding setup to enable in-situ welding training, enhancing the embodied learning of welding in three key ways.

  1. Visual XR Guides and Integrated Motion Sensing. Combined motion-sensing and visual XR feedback helps improve proprioceptive and embodied learning.
  2. Sensing Sonic Cues During Welding Practice. Acoustic sensing focuses learner attention on non-visual cues of weld performance.
  3. Mediated Meditation and Regulation. Biometric sensing enhances mindfulness and stress management in sensorially challenging environments
Simplified system diagram

‍

Contributions

The novelty of our work includes the ability for it to be used as an in-situ, functioning tool, as well as our holistic approach to augmentation.

Prior work in the areas of extended reality and welding create isolated, fully immersive simulations of welding, rather than an augmentation of actual welding in context. Existing products and research include full VR welding experiences (training without a functional helmet or torch, but a visual augmentation with hand tracking) and AR/computer vision-based simulations using tracking markers.

Our system is implemented to work with a live spark, in real welding contexts. Beyond training muscle memory, our system also uses additional sensors and the Quest's onboard sensors to promote good practice through feedback on mindful breathing, weld performance, and other attributes.

Device assemblies
Updated system hardware features various comfort improvements, including a counterbalanced helmet to help reduce neck strain

‍

Impact

We developed, tested, and deployed a mixed-reality set-up that provides interactive instructions and feedback to trainees, helps them achieve and explore creative structures and outcomes, and engages them in learning and retention within the training program.

We partnered with the Industrial Arts Workshop (IAW), a non-profit which focuses on training teenagers in the underserved community of Hazelwood through creative arts welding projects. This project will enable us in the future to democratize learning to transfer hands-on skills by creating immersive interfaces in many fields such as operating machines, woodworking, and physical computing, and to improve recruitment and retention of a skilled workforce. There is a need for such a training set-up for all ages and vocations.

Feedback from ongoing user testing

“I’ve been a welder for 35 years, and it’s still giving me feedback that would help me, because I’ve gotten into bad habits over the years. Even as an experienced welder, just for five minutes of use, it helps me.”
“I know I tend to lose my positioning towards the end of a weld, and the helmet accurately detects and points out this issue. This validation helps me understand where I need to improve.”
“There’s the way we think we weld, there’s the way other people think we weld. Then there’s a machine that will tell us the truth about the way we weld. That’s what I like about it. The machine tells you the truth. There’s no lying to it.”
“Not everyone (student) knows how to ask the right questions, or even feels comfortable to ask us. I think this tool would be more comfortable for them.”
“We have students with hearing difficulties. I think having this tool could benefit us by conveying our lessons through visual feedback rather than relying on verbal explanations as we used to.”

‍

Modified 3D-printed welding helmet and gun attachments

‍

Publications and Press

Details on our co-design process and hardware development, system setup, training specifics, related works, and citations

Augmenting Embodied Learning in Welding Training: The Co-Design of an XR- and tinyML-Enabled Welding System for Creative Arts and Manufacturing Training
‍ACM Tangible Embedded and Embodied Interaction 2024
Honorable Mention

Augmenting Welding Training: An XR Platform to Foster Muscle Memory and Mindfulness for Skills Development
‍ACM Interactive Surfaces and Spaces 2023
Best Demo

XRweld: An In-Situ Extended Reality Platform for Welding Education
ACM SIGGRAPH 2024

Machine learning and extended reality used to train welders
Feature: Carnegie Mellon University College of Engineering

Build Back Better Grant

Click to navigate to the top
A Changing Arctic, Carnegie Museum of Natural History
Augmented Learning Tools
Cara, CVS Pharmacy
Wishwell
Artwork
Music Programming
Hand-written name, "Ann Li"
InstagramTwitterLinkedinBehanceDribbble

Ann Li 2020

Don't be a stranger
If you think I'd be a good fit for your project or team, let's connect.
© 2024 Ann Li
LinkedIn link