Abstract

Prototyping use cases for augmented reality (AR) applications can be beneficial to elicit the functional requirements of the features early-on, to drive the subsequent development in a goal-oriented manner. Doing so would require designers to identify the goal-oriented interactions and map the associations between those interactions in a spatio-temporal context. Pertaining to the multiple scenarios that may result from the mapping, and the embodied nature of the interaction components, recent AR prototyping methods lack the support to adequately capture and communicate the intent of designers and stakeholders during this process. We present ImpersonatAR, a mobile-device-based prototyping tool that utilizes embodied demonstrations in the augmented environment to support prototyping and evaluation of multi-scenario AR use cases. The approach uses: (1) capturing events or steps in the form of embodied demonstrations using avatars and 3D animations, (2) organizing events and steps to compose multi-scenario experience, and finally (3) allowing stakeholders to explore the scenarios through interactive role-play with the prototypes. We conducted a user study with ten participants to prototype use cases using ImpersonatAR from two different AR application features. Results validated that ImpersonatAR promotes exploration and evaluation of diverse design possibilities of multi-scenario AR use cases through embodied representations of the different scenarios.

References

1.
Cockburn
,
A.
,
1997
, “
Structuring Use Cases With Goals
,”
J. Object-Oriented Program.
,
10
(
5
), pp.
56
62
.
2.
Kulak
,
D.
, and
Guiney
,
E.
,
2012
,
Use Cases: Requirements in Context
,
Addison-Wesley
,
Boston, MA
.
3.
Schneider
,
G.
, and
Winters
,
J. P.
,
2001
,
Applying Use Cases: A Practical Guide
,
Pearson Education
,
Boston, MA
.
4.
Duin
,
A. H.
,
Armfield
,
D. M.
, and
Pedersen
,
I.
,
2019
, “Human-Centered Content Design in Augmented Reality,”
Content Strategy in Technical Communication
,
G.
Getto
,
J.
Labriola
, and
S.
Ruszkiewicz
, eds.,
Routledge
,
Oxfordshire
, pp.
89
116
.
5.
Evans
,
K.
, and
Koepfler
,
J.
,
2017
, “
The UX of AR: Toward a Human-Centered Definition of Augmented Reality
,”
User Exp. Mag.
,
17
(
5
). https://uxpamagazine.org/the-ux-of-ar/
6.
Freitas
,
G.
,
Pinho
,
M. S.
,
Silveira
,
M. S.
, and
Maurer
,
F.
,
2020
, “
A Systematic Review of Rapid Prototyping Tools for Augmented Reality
,”
2020 22nd Symposium on Virtual and Augmented Reality (SVR)
,
Porto de Galinhas, Brazil
,
Nov. 7–10
, pp.
199
209
.
7.
Leiva
,
G.
,
Nguyen
,
C.
,
Kazi
,
R. H.
, and
Asente
,
P.
,
2020
, “
Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction
,”
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
,
Island of Oahu, HI
,
Apr. 25–30
, pp.
1
13
.
8.
Xu
,
Y.
,
Barba
,
E.
,
Radu
,
I.
,
Gandy
,
M.
,
Shemaka
,
R.
,
Schrank
,
B.
,
MacIntyre
,
B.
, and
Tseng
,
T.
,
2011
, “
Pre-Patterns for Designing Embodied Interactions in Handheld Augmented Reality Games
,”
2011 IEEE International Symposium on Mixed and Augmented Reality-Arts, Media, and Humanities
,
Basel, Switzerland
,
Oct. 26–29
, IEEE, pp.
19
28
.
9.
Webel
,
S.
,
Bockholt
,
U.
,
Engelke
,
T.
,
Gavish
,
N.
,
Olbrich
,
M.
, and
Preusche
,
C.
,
2013
, “
An Augmented Reality Training Platform for Assembly and Maintenance Skills
,”
Rob. Auton. Syst.
,
61
(
4
), pp.
398
403
.
10.
Johnson-Glenberg
,
M. C.
,
Birchfield
,
D. A.
,
Tolentino
,
L.
, and
Koziupa
,
T.
,
2014
, “
Collaborative Embodied Learning in Mixed Reality Motion-Capture Environments: Two Science Studies.
,”
J. Edu. Psychol.
,
106
(
1
), p.
86
.
11.
Shaghaghian
,
Z.
,
Burte
,
H.
,
Song
,
D.
, and
Yan
,
W.
,
2022
, “
Design and Evaluation of an Augmented Reality App for Learning Spatial Transformations and Their Mathematical Representations
,”
2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)
,
Christchurch, New Zealand
,
Mar. 12–16
, IEEE, pp.
608
609
.
12.
Shaghaghian
,
Z.
,
Burte
,
H.
,
Song
,
D.
, and
Yan
,
W.
,
2022
, “
An Augmented Reality Application and User Study for Understanding and Learning Spatial Transformation Matrices
,” preprint arXiv:2212.00110.
13.
Cao
,
Y.
,
Qian
,
X.
,
Wang
,
T.
,
Lee
,
R.
,
Huo
,
K.
, and
Ramani
,
K.
,
2020
, “
An Exploratory Study of Augmented Reality Presence for Tutoring Machine Tasks
,”
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
,
Honolulu, HI
,
Apr. 25–30
, pp.
1
13
.
14.
Genay
,
A.
,
Lécuyer
,
A.
, and
Hachet
,
M.
,
2021
, “
Being an Avatar ‘for Real’: A Survey on Virtual Embodiment in Augmented Reality
,”
IEEE Trans. Vis. Comput. Graph.
,
28
(
12
), pp.
5071
5090
.
15.
Alce
,
G.
,
Hermodsson
,
K.
, and
Wallergård
,
M.
,
2013
, “
WozARd: A Wizard of OZ Tool for Mobile AR
,”
Proceedings of the 15th International Conference on Human-Computer Interaction With Mobile Devices and Services, MobileHCI ’13
,
Munich, Germany
,
Aug. 27–30
, pp.
600
605
.
16.
Nebeling
,
M.
, and
Madier
,
K.
,
2019
, “
360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes From Paper
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
,
Glasgow, Scotland, UK
,
May 4–9
, pp.
1
13
.
17.
Nebeling
,
M.
,
Nebeling
,
J.
,
Yu
,
A.
, and
Rumble
,
R.
,
2018
, “
Protoar: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications
,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
,
Montreal, QC, Canada
,
Apr. 21–26
, pp.
1
12
.
18.
Leiva
,
G.
, and
Beaudouin-Lafon
,
M.
,
2018
, “
Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability
,”
Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST ’18
,
Berlin, Germany
,
Oct. 14–17
, pp.
675
682
.
19.
Hong
,
D.
,
2019
, “
Augmented Reality Prototyping Tools for Head-Mounted Displays
,” https://medium.com/@dylanhongtech/augmented-reality-prototyping-tools-for-head-mounted-displays-c1ee8eaa0783.
20.
Google
,
2021
, “
Tilt Brush
.”
21.
MacIntyre
,
B.
,
Gandy
,
M.
,
Dow
,
S.
, and
Bolter
,
J. D.
,
2004
, “
Dart: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences
,”
Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST ’04
,
Santa Fe, NM
,
Oct. 24–27
, pp.
197
206
.
22.
Kelly
,
A.
,
Shapiro
,
R. B.
,
de Halleux
,
J.
, and
Ball
,
T.
,
2018
, “
Arcadia: A Rapid Prototyping Platform for Real-Time Tangible Interfaces
,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
,
Montreal, QC, Canada
,
Apr. 21–26
, pp.
1
8
.
23.
Mackay
,
W. E.
,
2002
, “
Using Video to Support Interaction Design
,” DVD Tutorial, CHI, 2(5).
24.
Leiva
,
G.
,
Grønbæk
,
J. E.
,
Klokmose
,
C. N.
,
Nguyen
,
C.
,
Kazi
,
R. H.
, and
Asente
,
P.
,
2021
, “
Rapido: Prototyping Interactive AR Experiences Through Programming by Demonstration
,”
The 34th Annual ACM Symposium on User Interface Software and Technology
,
Virtual Event
,
Oct. 10–14
, pp.
626
637
.
25.
Billinghurst
,
M.
, and
Nebeling
,
M.
,
2022
, “
Rapid Prototyping of XR Experiences
,”
ACM SIGGRAPH 2022 Courses
,
Vancouver, BC, Canada
,
Aug. 7–11
, pp.
1
124
.
26.
Nebeling
,
M.
, and
Speicher
,
M.
,
2018
, “
The Trouble With Augmented Reality/Virtual Reality Authoring Tools
,”
2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)
,
Munich, Germany
,
Oct. 16–20
, IEEE, pp.
333
337
.
27.
Ye
,
H.
, and
Fu
,
H.
,
2022
, “
Progesar: Mobile AR Prototyping for Proxemic and Gestural Interactions With Real-World IoT Enhanced Spaces
,”
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
,
New Orleans, LA
,
Apr. 29–May 5
, pp.
1
14
.
28.
Cao
,
Y.
,
Xu
,
Z.
,
Li
,
F.
,
Zhong
,
W.
,
Huo
,
K.
, and
Ramani
,
K.
,
2019
, “
V.Ra: An In-Situ Visual Authoring System for Robot-IoT Task Planning With Augmented Reality
,”
Proceedings of the 2019 on Designing Interactive Systems Conference, DIS ’19
,
San Diego, CA
,
June 23–28
,
ACM
, pp.
1059
1070
.
29.
Cao
,
Y.
,
Wang
,
T.
,
Qian
,
X.
,
Rao
,
P. S.
,
Wadhawan
,
M.
,
Huo
,
K.
, and
Ramani
,
K.
,
2019
, “
GhostAR: A Time-Space Editor for Embodied Authoring of Human–Robot Collaborative Task With Augmented Reality
,”
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, UIST’19
,
New Orleans, LA
,
Oct. 20–23
,
ACM
, pp.
521
534
.
30.
Wang
,
T.
,
Qian
,
X.
,
He
,
F.
,
Hu
,
X.
,
Huo
,
K.
,
Cao
,
Y.
, and
Ramani
,
K.
,
2020
, “
CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications
,”
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST ’20
,
Virtual Event
,
Oct. 20–23
, pp.
328
341
.
31.
Rosson
,
M. B.
, and
Carroll
,
J. M.
,
2002
,
Usability Engineering: Scenario-Based Development of Human–Computer Interaction
,
Morgan Kaufmann
,
Burlington, MA
.
32.
Chi
,
P.-Y.
,
Li
,
Y.
, and
Hartmann
,
B.
,
2016
, “
Enhancing Cross-Device Interaction Scripting With Interactive Illustrations
,”
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
,
San Jose, CA
,
May 7–12
, pp.
5482
5493
.
33.
Liu
,
A. L.
, and
Li
,
Y.
,
2007
, “
Brickroad: A Light-Weight Tool for Spontaneous Design of Location-Enhanced Applications
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
San Jose, CA
,
Apr. 28–May 3
, pp.
295
298
.
34.
Li
,
Y.
,
Hong
,
J. I.
, and
Landay
,
J. A.
,
2004
, “
Topiary: A Tool for Prototyping Location-Enhanced Applications
,”
Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology
,
Santa Fe, NM
,
Oct. 24–27
, pp.
217
226
.
35.
Mangano
,
N.
,
LaToza
,
T. D.
,
Petre
,
M.
, and
van der Hoek
,
A.
,
2014
, “
Supporting Informal Design With Interactive Whiteboards
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Toronto, Ontario, Canada
,
Apr. 26–May 1
, pp.
331
340
.
36.
Tharatipyakul
,
A.
,
Lee
,
H.
,
Zhao
,
S.
, and
Davis
,
R. C.
,
2016
, “
Supporting the Comparison of Alternative Stories
,”
Proceedings of the 28th Australian Conference on Computer–Human Interaction
,
Launceston, Tasmania, Australia
,
Nov. 29–Dec. 2
, ACM, pp.
266
270
.
37.
Kim
,
H.-J.
,
Kim
,
C. M.
, and
Nam
,
T.-J.
,
2018
, “
Sketchstudio: Experience Prototyping With 2.5-Ddimensional Animated Design Scenarios
,”
Proceedings of the 2018 Designing Interactive Systems Conference
,
Hong Kong, China
,
June 9–13
, pp.
831
843
.
38.
Speicher
,
M.
,
Lewis
,
K.
, and
Nebeling
,
M.
,
2021
, “
Designers, the Stage Is Yours! Medium-Fidelity Prototyping of Augmented & Virtual Reality Interfaces With 360theater
,”
Proc. ACM Human–Comput. Interact.
,
5
(
EICS
), pp.
1
25
.
39.
Pettersson
,
J. S.
, and
Wik
,
M.
,
2014
, “
Perspectives on Ozlab in the Cloud: A Literature Review of Tools Supporting Wizard-of-OZ Experimentation, Including an Historical Overview of 1971–2013 and Notes on Methodological Issues and Supporting Generic tools
,”
Working Paper (2nd ed 2015), Department of Information Systems, Karlstad University
.
40.
Piumsomboon
,
T.
,
Lee
,
G. A.
,
Hart
,
J. D.
,
Ens
,
B.
,
Lindeman
,
R. W.
,
Thomas
,
B. H.
, and
Billinghurst
,
M.
,
2018
, “
Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration
,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18
,
Montreal QC Canada
,
Apr. 21–26
.
41.
Thoravi Kumaravel
,
B.
,
Anderson
,
F.
,
Fitzmaurice
,
G.
,
Hartmann
,
B.
, and
Grossman
,
T.
,
2019
, “
Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence
,”
Proceedings of the 32Nd Annual ACM Symposium on User Interface Software and Technology, UIST ’19
,
New Orleans, LA
,
Oct. 20–23
,
ACM
, pp.
161
174
.
42.
Yu
,
K.
,
Eck
,
U.
,
Pankratz
,
F.
,
Lazarovici
,
M.
,
Wilhelm
,
D.
, and
Navab
,
N.
,
2022
, “
Duplicated Reality for Co-Located Augmented Reality Collaboration
,”
IEEE Trans. Vis. Comput. Graph.
,
28
(
5
), pp.
2190
2200
.
43.
Maudet
,
N.
,
Leiva
,
G.
,
Beaudouin-Lafon
,
M.
, and
Mackay
,
W.
,
2017
, “
Design Breakdowns: Designer-Developer Gaps in Representing and Interpreting Interactive Systems
,”
Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’17
,
Portland, OR
,
Feb. 25–Mar. 1
, pp.
630
641
.
44.
Lee
,
G. A.
,
Kim
,
G. J.
, and
Billinghurst
,
M.
,
2005
, “
Immersive Authoring: What You Experience Is What You Get (wyxiwyg)
,”
Commun. ACM
,
48
(
7
), pp.
76
81
.
45.
Kim
,
Y.
, and
Bae
,
S.-H.
,
2016
, “
Sketchingwithhands: 3D Sketching Handheld Products With First-Person Hand Posture
,”
Proceedings of the 29th Annual Symposium on User Interface Software and Technology
,
Tokyo, Japan
,
Oct. 16–19
, pp.
797
808
.
46.
Gupta
,
A.
,
Agrawala
,
M.
,
Curless
,
B.
, and
Cohen
,
M.
,
2014
, “
Motionmontage: A System to Annotate and Combine Motion Takes for 3D Animations
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Toronto, Ontario, Canada
,
Apr. 26–May 1
, pp.
2017
2026
.
47.
Saquib
,
N.
,
Kazi
,
R. H.
,
Wei
,
L.-Y.
, and
Li
,
W.
,
2019
, “
Interactive Body-Driven Graphics for Augmented Video Performance
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
,
Glasgow, Scotland, UK
,
May 4 –9
, pp.
1
12
.
48.
Whitlock
,
M.
,
Fitzmaurice
,
G.
,
Grossman
,
T.
, and
Matejka
,
J.
,
2020
, “
AuthAR: Concurrent Authoring of Tutorials for AR Assembly Guidance
,”
Graphics Interface 2020
,
Toronto, ON, Canada
,
May 28–29
.
49.
Chidambaram
,
S.
,
Huang
,
H.
,
He
,
F.
,
Qian
,
X.
,
Villanueva
,
A. M.
,
Redick
,
T. S.
,
Stuerzlinger
,
W.
, and
Ramani
,
K.
,
2021
, “
Processar: An Augmented Reality-Based Tool to Create In-Situ Procedural 2D/3D AR Instructions
,”
Designing Interactive Systems Conference 2021
,
Virtual Event
,
June 28–July 2
, pp.
234
249
.
50.
Vogt
,
D.
,
Stepputtis
,
S.
,
Grehl
,
S.
,
Jung
,
B.
, and
Amor
,
H. B.
,
2017
, “
A System for Learning Continuous Human–Robot Interactions From Human–Human Demonstrations
,”
2017 IEEE International Conference on Robotics and Automation (ICRA)
,
Marina Bay Sands, Singapore
,
May 29–June 3
, IEEE, pp.
2882
2889
.
51.
Porfirio
,
D.
,
Fisher
,
E.
,
Sauppé
,
A.
,
Albarghouthi
,
A.
, and
Mutlu
,
B.
,
2019
, “
Bodystorming Human–Robot Interactions
,”
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology
,
New Orleans, LA
,
Oct. 20–23
, pp.
479
491
.
52.
,
H.
, and
Li
,
Y.
,
2013
, “
Gesture Studio: Authoring Multi-Touch Interactions Through Demonstration and Declaration
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Paris, France
,
Apr. 27–May 2
, pp.
257
266
.
53.
Dey
,
A. K.
,
Hamid
,
R.
,
Beckmann
,
C.
,
Li
,
I.
, and
Hsu
,
D.
,
2004
, “
A Cappella: Programming by Demonstration of Context-Aware Applications
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Vienna, Austria
,
Apr. 24–29
, pp.
33
40
.
54.
Hartmann
,
B.
,
Abdulla
,
L.
,
Mittal
,
M.
, and
Klemmer
,
S. R.
,
2007
, “
Authoring Sensor-Based Interactions by Demonstration With Direct Manipulation and Pattern Recognition
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
San Jose, CA
,
Apr. 28–May 3
, pp.
145
154
.
55.
Kim
,
J.-W.
,
Kim
,
H.-J.
, and
Nam
,
T.-J.
,
2016
, “
M. Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices
,”
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
,
San Jose, CA
,
May 7–12
, pp.
2307
2318
.
56.
Ashbrook
,
D.
, and
Starner
,
T.
,
2010
, “
Magic: A Motion Gesture Design Tool
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Atlanta, GA
,
Apr. 10–15
, pp.
2159
2168
.
57.
Wang
,
T.
,
Qian
,
X.
,
He
,
F.
,
Hu
,
X.
,
Cao
,
Y.
, and
Ramani
,
K.
,
2021
, “
Gesturar: An Authoring System for Creating Freehand Interactive Augmented Reality Applications
,”
The 34th Annual ACM Symposium on User Interface Software and Technology
,
Virtual Event
,
Oct. 10–14
, pp.
552
567
.
58.
Chua
,
P. T.
,
Crivella
,
R.
,
Daly
,
B.
,
Hu
,
N.
,
Schaaf
,
R.
,
Ventura
,
D.
,
Camill
,
T.
,
Hodgins
,
J.
, and
Pausch
,
R.
,
2003
, “
Training for Physical Tasks in Virtual Environments: Tai Chi
,”
Proceedings of the IEEE Virtual Reality, 2003
,
Los Angeles, CA
,
Mar. 22–26
, IEEE, pp.
87
94
.
59.
Anderson
,
F.
,
Grossman
,
T.
,
Matejka
,
J.
, and
Fitzmaurice
,
G.
,
2013
, “
Youmove: Enhancing Movement Training With an Augmented Reality Mirror
,”
Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology
,
St. Andrews Scotland, UK
,
Oct. 8–11
, pp.
311
320
.
60.
Piumsomboon
,
T.
,
Lee
,
G. A.
,
Irlitti
,
A.
,
Ens
,
B.
,
Thomas
,
B. H.
, and
Billinghurst
,
M.
,
2019
, “
On the Shoulder of the Giant: A Multi-scale Mixed Reality Collaboration With 360 Video Sharing and Tangible Interaction
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
,
Glasgow, Scotland, UK
,
May 4–9
, pp.
1
17
.
61.
Xu
,
W.
,
Liang
,
H.-N.
,
Zhao
,
Y.
,
Yu
,
D.
, and
Monteiro
,
D.
,
2019
, “
DMove: Directional Motion-Based Interaction for Augmented Reality Head-Mounted Displays
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19
,
Glasgow, Scotland, UK
,
May 4–9
, pp.
1
14
.
62.
Edwards
,
R.
,
Bybee
,
B. T.
,
Frost
,
J. K.
,
Harvey
,
A. J.
, and
Navarro
,
M.
,
2017
, “
That’s Not What I Meant: How Misunderstanding Is Related to Channel and Perspective-Taking
,”
J. Lang. Soc. Psychol.
,
36
(
2
), pp.
188
210
.
63.
Jacob
,
R. J.
,
Girouard
,
A.
,
Hirshfield
,
L. M.
,
Horn
,
M. S.
,
Shaer
,
O.
,
Solovey
,
E. T.
, and
Zigelbaum
,
J.
,
2008
, “
Reality-Based Interaction: A Framework for Post-Wimp Interfaces
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Florence, Italy
,
Apr. 5–10
, pp.
201
210
.
64.
Fiorella
,
Logan
,
2021
, “23 the Embodiment Principle in Multimedia Learning,”
The Cambridge Handbook of Multimedia Learning
, 3rd ed.,
R. E.
Mayer
, and
L.
Fiorella
, eds.,
Cambridge University Press
,
Cambridge
, pp.
286
295
.
You do not currently have access to this content.