Tutorial: Using and Adapting Human-Computer Interaction Evaluation Methods for HRI

December 5, 2008 – 9:06 am

Participants will learn about human-computer interaction (HCI) evaluation methods that have been used and adapted for HRI.  This tutorial will cover examples from all three categories of HCI evaluation methods—inspection, empirical, and formal/analytical. Further, attendees will learn what type of evaluation technique(s) and metrics are best suited to different goals and situations, taking into account the unique challenges of evaluating robot interaction. Lecture and discussion will be interspersed with hands-on tasks in which groups of participants will evaluate a robot interface.  This course is designed to complement and supplement the course given by Dr. Greg Trafton at previous HRI conferences, Experimental Design for HRI.

2. Background of participants

This course assumes a familiarity of evaluating HRI such as was provided in earlier HRI tutorials or experience with basic experimental evaluation techniques.  Participants should be familiar with human-robot interaction but we will not assume prior knowledge of specific HCI evaluation techniques.

3. Motivation

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

As HRI becomes a more mature discipline, expectations are rising for validating the work that we are doing to design robot interfaces.  HRI is its own sub-discipline, but there are similarities to HCI because robots are computerized applications — even though they are usually mobile, often remote from users, sometimes fragile, and potentially unpredictable in their behavior.  Rather than start from scratch, a number of researchers have taken HCI evaluation techniques and have adapted them to be more compatible with the practical realities of robots.  This tutorial describes several example techniques and their adaptations and provides guidance for using these techniques with HRI applications.

The instructors are active in the field of search and rescue robotics, assistive robotics, and unmanned aerial vehicles and will use examples from these domains in the presentations.  Having examples from multiple domains will enrich the discussion.

4. Conducting the tutorial

The tutorial will use lectures and exercises to give participants experience with using four example techniques.  The tutorial will begin with a lecture and discussion of how HRI differs from HCI and what roles robot users take on when working with robots (Scholtz 2003).  We will discuss situation awareness (SA) measurement techniques (e.g., Endsley’s (1998) Situation Awareness Global Assessment Technique) and which ones are appropriate for various types of robot evaluation tasks. The heart of the tutorial consists of modules on each of the three categories of HCI evaluation methods (inspection, empirical, and formal/ analytical) and exercises that illustrate them, as described below.

4.1 Inspection Method: GDTA

Goal Directed Task Analysis (GDTA) was used by Adams (2005) to better understand how users will want to employ a robot interface.  A GDTA analysis can also be used as a means of inspecting an interface to evaluate whether it allows users to meet their goals efficiently.  We will provide attendees with a fragment of a previously-developed GDTA that they will use to evaluate whether the interface is compatible with the identified goals.

4.2 Inspection Method: Heuristic Evaluation

Heuristic evaluation for HRI (Drury et al. 2003) is an inspection method based on Nielsen’s (1994) technique, but modified with heuristics that apply more directly to HRI.  Attendees will practice comparing the heuristics to the example interface, identifying the parts of the interface that violate the heuristics.

4.3 Empirical Method: LASSO

The LASSO method (Drury et al. 2007a) is based on usability testing and the Think Aloud method (Ericsson and Simon 1980).  Users perform typical tasks while thinking aloud, and evaluators analyze the users’ utterances to determine whether users had awareness of the robot’s Location, Activity, Surroundings, and Status as well as progress towards completing the Overall mission (LASSO).  We will show a video clip from a LASSO evaluation and attendees will practice coding a transcript fragment.

4.4 Formal/Analytical Method: GOMS

The Goals, Operators, Methods and Selection rules (GOMS) technique (John and Kieras 1996) is a formal/analytical method.  In conjunction with Kieras, we have extended GOMS for HRI (Drury et al. 2007b).  We will guide attendees in writing a fragment of a GOMS model using the simplest variant of GOMS, the Keystroke Level Model, for the example robot interface.

5. Tutorial schedule

If there is more time available than is scheduled below, we will allow more time for exercises and discussion.

Activity	Format	Time
1.    Introduction •    Why HRI is different from HCI •    How HRI roles affect evaluation •    Evaluation and levels of autonomy	Lecture, videos	25 min.
2.    SA measurement techniques •    Types of SA measurement methods •    Matching methods to evaluation situations	Lecture, videos	35  min.
3.    Exercise 1: SA analysis using LASSO	Hands-on	30 min.
4.    Inspection evaluation techniques •    The difference between standards, guidelines, checklists, & heuristics	Lecture	20 min.
5.    Exercise 2: Goal Directed Task Analysis	Hands-on	30 min.
6.    Inspection evaluation techniques (continued) •    Tailoring heuristics for HRI	Lecture	10 min.
7.    Exercise 3:  Heuristic evaluation for HRI	Hands-on	30 min.
8.    Empirical techniques •    Usability testing •    Coding video & extracting metrics	Lecture, video	30 min.
9.    Exercise 4: Activity coding	Hands-on	30 min.
10. Formal/analytical techniques •    The GOMS variants and which ones are appropriate for different evaluation goals and tasks	Lecture, discussion	30 min.
11. Exercise 5: GOMS	Hands-on	30 min.
12. Wrap-up	Discussion	15 min.

6. Instructors’ backgrounds

Dr. Jill Drury is an Associate Department Head at The MITRE Corporation, Adjunct Assistant Professor of Computer Science at the University of Massachusetts Lowell, and Visiting Scientist at MIT.  Her research interests include human-robot interaction, evaluation methods for human-computer interaction, and awareness support for collaborative applications.  She regularly teaches Evaluation of HCI at the graduate level.  She is a member of ACM, SIGCHI, and SWE (Society of Women Engineers) and has had organizing committee and reviewer roles for many conferences, including HRI and Human Factors in Computing Systems (CHI).  She co-organized a tutorial with Dr. Yanco and Dr. Jean Scholtz on Introduction to HRI at the Intelligent User Interfaces (IUI) 2006 conference.  Her publications in the area can be found at jldrury.googlepages.com.

Dr. Holly Yanco is an Associate Professor of Computer Science at the University of Massachusetts Lowell.  Her research interests include human-robot interaction, artificial intelligence for robotics, assistive technology, and urban search and rescue.  She has doctorate and master’s degrees from the Massachusetts Institute of Technology and a bachelor’s degree from Wellesley College, all in Computer Science.  She received the Award for Teaching Excellence from the University of Massachusetts Lowell in 2002 and the Frederick C. Hennie III Teaching Award from the Electrical Engineering and Computer Science Department of the Massachusetts Institute of Technology in 1996.  She is an elected member of AAAI’s Executive Council and has served as the Exhibitions and Sponsorship Chair of HRI-07, HRI-08 and HRI-09.  With Dr. Jean Scholtz, she co-organized a tutorial on Introduction to HRI at the CHI 2004 and CHI 2005 conferences, and (also with Jill Drury) at the IUI 2006 conference.  She also presented a tutorial introducing HRI to the AAAI 2008 conference.  See a list of her publications at https://robotics.cs.uml.edu/publications/.

7. References

[1]    Adams, J. A. (2005). Human-Robot Interaction Design: Understanding User Needs and Requirements.  In Proceedings of the 2005 Human Factors and Ergonomics Society 49th Annual Meeting, 2005, Orlando, FL, USA.
[2]    Drury, J., Riek, L. D., Christiansen, A. D., Eyler-Walker, Z. T., Maggi, A. and Smith, D. B. (2003).  Command and Control of Robot Teams.  In Proceedings of the 2003 Association of Unmanned Vehicle Systems International, Baltimore, July 2003.
[3]    Drury, J. L., Keyes, B., and Yanco, H. A. (2007a).  LASSOing HRI: Analyzing Situation Awareness in Map-Centric and Video-Centric Interfaces.  In Proceedings of the Second Annual Conference on Human-Robot Interaction, Arlington, VA, March 2007.
[4]    Drury, J. L., Scholtz, J., and Kieras, D. E. (2007b).  Adapting GOMS to Model Human-Robot Interaction. In Proceedings of the Second Annual Conference on Human-Robot Interaction, Arlington, VA, March 2007.
[5]    Endsley, M. R., Selcon, S. J., Hardiman, T. D., and Croft, D. G. (1998).  A Comparative Analysis of SAGAT and SART for Evaluations of Situation Awareness.  In Proceedings of the 42nd annual meeting of the Human Factors and Ergonomics Society, Chicago, October 1998.
[6]    K. A. Ericsson and H. A. Simon (1980). Verbal Reports as Data. Psychological Review, Vol. 87, pp. 215 – 251.
[7]    John, B. E. and Kieras, D. E. (1996).  The GOMS Family of User Interface Analysis Techniques:  Comparison and Contrast.  ACM Transactions on Human-Computer Interaction, 3(4), December 1996.
[8]    Nielsen, J. (1994). Heuristic Evaluation.  In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods.  John Wiley & Sons, New York, NY.
[9]    Scholtz, J. (2003).  Theory and Evaluation of Human Robot Interactions.  In Proceedings of the Hawaii International Conference on System Science 36, January 2003